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11th  Annual  CMMI  Technology  Conference  and  User  Group 
Denver,  Co 
14  -17  November  2011 


Agenda 


Tuesday.  November  15.  2011 


MONDAY  TUTORIAL  SESSIONS 


TRACK  2  -  WIND  STAR 

•  13304  -  Making  Process  Improvement  Work  -  Tying  Improvement  and  CMMI®  Directly  to  What  You  Care  About,  Mr.  Neil  Potter,  The  Process  Group 

•  13555  -  Architecture:  Why  Your  CMMI®  VI. 3  Implementation  is  Incomplete  Without  It!,  Dr.  Lawrence  Jones  &  Dr.  Michael  Konrad,  Software 
Engineering  Institute 


TRACK  3  -  HIGHLANDS 

•  13492  -  Lesson  Learned  from  Pilot  Implementation  of  Organizational  Performance  Management  (OPM)  Process  Area,  Mr.  Kobi  Vider,  K.V.P. 
Consulting 

•  13493  -  Leveraging  Your  Service  Quality  Using  ITIL  V3,  ISO  20000  and  CMMI®-SVC,  Mr.  Kobi  Vider,  K.V.P.  Consulting 


TRACK  4  -  WIND  RIVER 

•  1343 1  -  Risk  and  Issue  Management,  Mr.  A1  Florence,  MITRE 

•  13479  -  Software  Safety,  Mr.  Tim  Kasse,  National  Renewable  Energy  Laboratory  (NREL) 


Tuesday.  November  15.  2011 


Keynote  Speaker 

•  Maj  Gen  Paul  Neilsen,  USAF  (Ret),  Director,  Software  Engineering  Institute 


Future  of  CMMI®  -  Panel  Discussion 

•  Mr.  Brian  Gallagher,  Northrop  Grumman  Corporation 
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•  Mr.  Rusty  Young,  Software  Engineering  Institute 


CONCURRENT  SESSIONS 


TRACK  1  -  MESA  VERDE 

•  13488  -  Just  Getting  Started  with  CMMI®,  Ms.  Mary  Beth  Chrissis,  Software  Engineering  Institute. 

•  13481  -  Are  Your  Customers  Happy?  Using  Customer  Satisfaction  to  Drive  Improvement  Efforts,  Ms.  Susan  LaFortune,  Department  of  Defense 

•  135 10  -  Lessons  Learned  in  Overcoming  Resistance  to  CMMI  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  13456  -  Diagnosis  PIF:  Avoiding  Process  Improvement  Fatigue,  Mr.  Craig  Hale,  Esterline  Control  Systems  -  AVISTA 

TRACK  2  -  WIND  STAR 

•  13427  -  Why  Do  Developers  Make  These  Dangerous  Software  Errors?,  Mrs.  Michele  Moss,  Booz  Allen  Hamilton 

•  13365  -  Some  Assembly  Required:  Using  Agile  Methodologies  to  Develop  an  Interactive  Software  User’s  Guide,  Mr.  Ronald  Stauffer,  Raytheon 
Company 

TRACK  3  -  HIGHLANDS 

•  12936  -  Applying  CMMPD-SVC  Process  Areas  as  an  Extension  to  CMMPD-DEV,  Ms.  Mary  Lynn  Penn,  Lockheed  Martin  IS&GS 

•  13263  -  A  LEAN  and  RACI  Approach  to  CMMI®  for  Services  (CMMI®-SVC),  Mr.  Ahn  Nuzen,  SPAWAR  Systems  Center  Pacific  (SSC  PAC) 

•  13291  -  A  Real-Life  Example  of  Appraising  and  Interpreting  CMMI®-  Services  Maturity  Level  2,  Mr.  Neil  Potter,  The  Process  Group 

•  13520  -  CMMPD-DEV:  Where  “Build  Stuff’  Happens  in  CMMI®-SVC,  Ms.  Eileen  Forrester,  Software  Engineering  Institute 

TRACK  4  -  WIND  RIVER 

•  13288  -  Comparing  Scrum  And  CMMI®  -  How  Can  They  Work  Together?,  Mr.  Neil  Potter,  The  Process  Group 

•  13446  -  Utilizing  Lean  Six  Sigma  to  Attain  CMMI®  Maturities,  Mr.  Steven  Moffat,  SAIC 

•  13465  -  Design  Your  Business  Processes  to  Embrace  People  in  an  Agile  Approach  and  Support  High  Maturity  (OPM),  Mr.  Kobi  Vider,  K.V.P. 
Consulting 


Wednesday.  November  16.  2011 


CONCURRENT  SESSIONS 


TRACK  1  -  MESA  VERDE 

•  13514  -  The  Fundamental  Value  of  Every  Single  CMMI®  Practice,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  13475  -  Questions  from  the  Trenches:  What  Over  1000  Students  Want  to  Know  Most  About  the  CMMI®,  Mr.  Bill  Smith,  Leading  Edge  Process 
Consultants 

•  13450  -  CMMI®  Training  -  Avoiding  Waste  and  Ineffectiveness,  Ms.  Sari  Schneider,  The  Boeing  Company 

•  13498  -  Communication  Loops  in  CMMI®  v.1.3,  Mr.  Shawn  Presson,  TestPros,  Inc. 

•  13472  -  Why  Project  Managers  (Understandably)  Hate  the  CMMI®  and  What  to  Do  About  It,  Mr.  Bill  Smith,  Leading  Edge  Process  Consultants 

•  13495  -  CMMI®  Level  1  to  3  in  15  Months:  AIM  for  a  Performance  Upgrade,  Dr.  Gene  Miluk,  Software  Engineering  Institute,  Carnegie  Mellon 
University 

•  13458  -  Project  Driven  Process  Improvement:  A  large  Scale  Lean  Six  Sigma  Deployment  in  Egypt  Across  Multiple  Small  and  Medium  Sized  Software 
Organizations,  Dr.  Radouane  Oudrhiri,  Systonomy 

TRACK  2  -  WIND  STAR 

•  13318  -  Chutes  and  Ladders  -  CMMI®  -  ISO  Considerations,  Ms.  Kimberley  Eley,  ManTech 

•  13477  -  A  Tale  of  Two  Cultures  Ms.  Beth  Layman,  Layman  &  Layman 

•  13303  -  Using  Lessons  Learned  from  Medical  Checklists  to  Simplify  CMMI®  Processes,  Mr.  Neil  Potter,  The  Process  Group 

•  13480  -  Creating  an  SQA  Program  at  the  National  Renewable  Energy  Laboratory  (NREL)  -  A  DOE  FFRDC  Based  on  the  NDIA/DoD  Sponsored 
CMMI®,  Mr.  Tim  Kasse,  National  Renewable  Energy  Laboratory 

•  13496  -  TSP  and  Architecture  in  the  Real  World,  Mr.  James  McHale,  Software  Engineering  Institute 

•  13433  -  Risk  Management  Mr.  A1  Florence,  The  MITRE  Corporation 

•  1351 1  -  Effective,  CMMI-Compliant  Project  Plans  (in  Less  Than  10  Pages),  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

TRACK  3  -  HIGHLANDS 

•  13459  -  A  Process  Performance  Models  Case  Study,  Mr.  Kobi  Vider,  K.V.P.  Consulting 

•  13500  -  Use  Causal  Analysis  and  Resolution  (CAR)  &  Quantitative  Project  Management  (QPM)  to  Monitor  and  Control  the  Quality  and  Process- 
Performance,  Mr.  Yasir  Madi,  OST  Global 

•  13508  -  Strategies  for  Retaining  CMMI®  Maturity  Level  5  in  vl.3,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  13516  -  Assembling  a  Multi-  Model  Approach  to  Improving  Service  Quality  and  Ensuring  Service  Resilience  in  Complex  Risk  Environments,  Ms. 
Eileen  Forrester,  Software  Engineering  Institute 
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12935  -  CMMI®  for  Acquisition  (CMMI®-ACQ)/CMMI®  for  Services  (CMMI®-ACQ)  and  Their  Potential  Contribution  on  the  Rapid  Acquisition  of  IT 
Systems  in  Support  of  Public  Law  1 1 1,  Dr.  Kenneth  Nidiffer,  Software  Engineering  Institute 

•  13452  -  Implementation  of  Process  Improvement  in  a  Multi-Model/Standard  Government  Environment,  Mr.  Nathaniel  Becker,  U.S.  Army  ARDEC 

TRACK  4  -  WIND  RIVER 

•  13290  -  Appraisals  and  CMMI®  Gotchas  -  Lessons  in  CMMI®  Use  and  Appraisal  Preparation,  Mr.  Neil  Potter,  The  Process  Group 

•  13451  -  Streamlining  Process  Deployment  and  Compliance,  Mr.  Gary  Natwick,  Harris  Corporation 

•  135 12  -  How  to  Successfully  and  Cost-Effectively  Conduct  a  Re- Appraisal,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  13240  -  CMMI®  Maturity  Level  5  A  Bargain!,  Mrs.  Jayne  Perkins,  Raytheon  IDS 

•  13402  -  CMMI®  Surveillance  Appraisals:  A  Modest  Proposal,  NDIA  CMMI®Working  Group 

•  12987  -  SCAMPI  Planning  with  vl  .3, Ms.  Doma  Witkowski,  Lockheed  Martin  Corporation 

•  13506  -  Increasing  Efficiency  and  Saving  Costs  with  New  SCAMPI  Approaches,  Mr.  Paul  Byrnes,  ISD,  Inc. 

•  13485  -  Experiences  in  the  Applying  of  MDD  (Method  Definition  Document)  vl.3  Sampling  Factors  and  Sampling  Algorithm,  Mr.  Bradley  Bittorf, 
Raytheon  Company 


Thursday.  November  17.  2011 


CONCURRENT  SESSIONS 


TRACK  1  -  MESA  VERDE 

•  13644  -  CMMI  for  Executives,  NDIA  CMMI®Working  Group 

•  13471  -  Cracking  the  Code  for  Improving  the  Productivity  of  Knowledge  Workers,  Mr.  Peter  Voldby  Petersen,  Callis 

•  13455  -  Improving  Process  Quality  While  Lowering  Costs,  Mr.  Craig  Hale,  Esterline  Control  Systems  -  AVISTA, 

•  13505  -  The  Change  Comes  from  the  Inside!  Divide  and  Conquer:  A  Top  Down  and  Bottom  Up  Approach,  Mr.  Surya  Kanchiraju,  OST,  Inc. 

TRACK  2  -  WIND  STAR 

•  13028  -  Process  Acceptance  Through  Use  Cases,  Dr.  Juergen  Schmied,  Method  Park 

•  13482  -  How  a  Focus  on  High  Maturity  CMMI®-Based  Process  Improvement  Can  Add  Value  to  the  Organizations  Even  When  There  are  Only  Limited 
Resources,  Mr.  Edmond  Sung,  Processis,  LTD 

•  12962  -  Parametric  Estimation  for  ERP  Implementations,  Mr.  Donald  Beckett,  Quantitative  Software  Management 

•  13434  -  Writing  Requirements  Properly,  Mr.  A1  Florence,  The  MITRE  Corporation 

TRACK  3  -  HIGHLANDS 

•  13491  -  CMMI®,  ISO  and  AS9100,  An  Efficient  and  Effective  Approach,  Mrs.  LaKeisha  Souter,  Northrop  Grumman  Corporation 

•  13468  -  Using  the  SEI  Models  and  Practices  to  Assure  your  Subcontractors  quality  with  Cross  Constellations  and  Multi-Models  Inspiration,  Mr.  Kobi 
Vider,  K.V.P.  Consulting 

•  13460  -  Lean  Six  Sigma,  Towards  an  Empirical  and  Experimental  Approach  to  Software  Process,  Dr.  Radouane  Oudrhiri,  Systonomy 

•  13419  -  Raytheon  Pasadena  Operations,  CMMI®-DEV  versus  CMMI®-SVC  Analysis,  Mrs.  Rose-Marie  Gonzalez,  Raytheon 

TRACK  4  -  WIND  RIVER 

•  13449  -CMMI®  Without  Appraisals  Mr.  Alan  Gellis,  The  Boeing  Company 

•  13643  -  SCAMPI  A  VI  .3  MDD  Usage  and  Profile,  Mr.  Michael  Campo,  Raytheon  Company 

•  13421  -  You  are  a  CMMI®-DEV  Appraisal  Expert?  What  Do  You  Do  If  Your  Organization  Wants  to  Do  a  CMMI®-SVC  Appraisal?,  Mr.  Joseph 
Trujillo,  Raytheon 

•  13404  -  Managed  Discovery?  PIIDs?  How  Do  I  Decide  Which  SCAMPI  Data  Collection  Techniques  are  Right  for  My  Organization,  Mr.  Sam  Fogle, 
ACE  Guides 

2011  CMMI-CREATE  Survey 
2011  CMMI-CREATE  Survey  Results 
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CMML 

11™  ANNUAL 

CMMI®  TECHNOLOGY 
CONFERENCE  AND 
USER  GROUP 


Who  Should  Attend? 


Defense,  aerospace  and 
commercial  companies,  CMMI® 
Transition  Partners,  Department 
of  Defense  organizations,  small 
companies  specializing  in 
software  and  systems  engineering 
development,  tools  and  processes, 
acquisition,  or  services,  and  other 
government  agencies. 


What  Will  Be  Presented? 


A  wide  variety  of  presentations 
including  the  new  CMMI® 
for  Services,  integrated  process 
improvement,  Lean/Agile  and  Six 
Sigma  approaches,  and  evolving 
approaches  and  lessons  learned 
involving  SCAMPI  (SM)  appraisal 
methods. 


NOVEMBER  14-1 7, 2011 


SPONSOREDgYTHE  NDIA  SYS 

'""'-J^^NOYTECH  CENTER 


WWW.NDIA.ORG/MEETINGS/21 10 


EVENT  #2110 
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SUNDAY,  NOVEMBER  13, 2011 

4:00  pm  -  6:00  pm  Registration  Open 

Located  in  Grand  Mesa  Foyer,  2ndFloor 


MONDAY,  NOVEMBER  14, 2011 

7:00  am  -  6:00  pm  Registration  Open 

Located  in  Grand  Mesa  Foyer,  2ndFloor 

7:00  am  -  8:00  am  Continental  Breakfast  (Tutorial  Attendees  Only) 


Located  in  Grand  Mesa  Foyer,  2nd Floor 


8:00  am  -  5:00  pm 
9:45  am -10:15  am 
11:45  am -1:00  pm 
2:45  pm -3:15  pm 
5:00  pm  -  6:00  pm 


Tutorial  Sessions  (Tutorial  Attendees  Only;  Additional  Cost 
Required) 

Break  (Tutorial  Attendees  Only) 

Located  in  Grand  Mesa  Foyer,  2ndFloor 

Lunch  (Tutorial  Attendees  Only) 

Located  in  Grand  Mesa  ABC,  2"dFloor 

Break  (Tutorial  Attendees  Only) 

Located  in  Grand  Mesa  Foyer,  2ndFloor 

Reception  (Open  to  ALL  ATTENDEES) 


Located  in  Atrium  Display  Area,  2nd  Floor 


TUESDAY,  NOVEMBER  15, 2011 

7:00  am  -  6:00  pm  Registration  Open 

Located  in  Grand  Mesa  Foyer ;  2nd Floor 

7:00  am  -  8:00  am  Continental  Breakfast 

Located  in  Atrium  Display  Area,  2nd Floor 

8:00  am  -  8:15  am  Welcome  and  Opening  Remarks 

Located  in  Mesa  Verde,  2nd  Floor 

►  Mr.  Sam  Campagna,  Assistant  Vice  President,  Operations,  NDIA 

►  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon 
Company 


8:1 5  am  -  9:1 5  am  Keynote  Speaker 

Located  in  Mesa  Verde,  2ndFloor 

►  Maj  Gen  Paul  Neilsen,  USAF  (Ret),  Director,  Software 
Engineering  Lnstitute 


9:15  am  - 10:00  am  State  of  CMMI® 

Located  in  Mesa  Verde,  2ndFloor 

►  Mr.  Mike  Phillips,  Software  Engineering  Institute 

10:00  am  -  10:30  am  Break 

Located  in  Atrium  Display  Area,  2ndFloor 

10:30  am  - 12:00  pm  Future  of  CMMI®  -  Panel  Discussion 

Located  in  Mesa  Verde,  2ndFloorr 

►  Mr.  Michael  Campo,  Raytheon  Company 

►  Mr.  Brian  Gallagher,  Northrop  Grumman  Corporation 

►  Ms.  Lynn  Penn,  Lockheed  Martin  Corporation 

►  Mr.  Rusty  Young,  Software  Engineering  Lnstitute 
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12:00  pm  -1:30  pm 

Luncheon  Speaker 

Located  in  Grand  Mesa  ABC,  2nd Flo  or 
“ What  Do  You  Know  about  Process  Improvement  and 

Physics-Based  Modeling  ” 

►  Mr.  Geoff  Draper,  Harris  Corporation 

1 :30  pm  -  3:00  pm 

Concurrent  Sessions 

3:00  pm  -  3:30  pm 

Break 

Located  in  Atrium  Display  Area,  2nd Floor 

3:30  pm  -  5:00  pm 

Concurrent  Sessions 

5:00  pm  -  6:00  pm 

Reception 

Located  in  Atrium  Display  Area,  2nd Floor 

WEDNESDAY,  NOVEMBER  16, 2011 

7:00  am  -  5:00  pm  Registration  Open 


7:30  am  -  8:30  am 

Located  in  Grand  Mesa  Foyer,  2ndFloor 

Continental  Breakfast 

Located  in  Atrium  Display  Area,  2nd Floor 

8:30  am  - 10:00  am 

Concurrent  Sessions 

10:00  am -10:30  am  Break 

Located  in  Atrium  Display  Area,  2nd Floor 

10:30  am  - 12:00  pm  Concurrent  Sessions 
1 2:00  am  - 1 :30  pm  Luncheon 


1:30  pm  -3:00  pm 

Located  in  Grand  Mesa  ABC,  2nd Floor 

Concurrent  Sessions 

3:00  pm  -  3:30  pm 

Break 

Located  in  Atrium  Display  Area,  2nd Floor 

3:30  pm  -  5:00  pm 

Concurrent  Sessions 

5:00  pm 

Conference  Ajourns  for  the  Day 

THURSDAY,  NOVEMBER  1 7, 201 1 

7:00  am  - 12:00  am  Registration  Open 


7:30  am  -  8:30  am 

Located  in  Grand  Mesa  Foyer,  2ndFloor 

Continental  Breakfast 

Located  in  Grand  Mesa  Foyer,  2"dFloor 

8:30  am  - 10:00  am 

Concurrent  Sessions 

10:00  am -10:30  am  Break 

Located  in  Grand  Mesa  Foyer,  2"dFloor 

10:30  am  - 12:00  pm  Concurrent  Sessions 
1 2:00  pm  Conference  Ajourns 

**Please  note  that  the  agenda  is  subject  to  change** 
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MONDAY,  NOVEMBER  14, 2011 
TUTORIAL  SESSIONS 


Track  1 

Track  2 

Track  3 

Track  4 

Mesa  Verde 

Wind  Star 

Highlands 

Wind  River 

8:00  am 

9:45  am 

Session  A 

1A1  -  Tutorial 

1 3435  -  How  to  Design  Lean 

CMMI®  Processes  and  Procedures 
Using  Architectures  and  Models 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

1A2  -  Tutorial 

13304  -  Making  Process 

Improvement  Work  -  Tying 
Improvement  and  CMMI®  Directly  to 
What  You  Care  About 

Mr.  Neil  Potter,  The  Process  Group 

1  A3  -  Tutorial 

13492  -  Lesson  Learned  from  Pilot 
Implementation  of  Organizational 
Performance  Management  (0PM) 
Process  Area 

Mr.  Kobi  Vider,  K.  1 IP.  Consulting 

1A4  -  Tutorial 

13431  -  Risk  and  Issue 

Management 

Mr.  Al  Florence,  MITRE 

Break  in  Grand  Mesa  Foyer,  2nd  Floor 

10:15  am 

11:45  am 

Session  B 

1B1  -  Tutorial  (Continued) 

1 3435  -  How  to  Design  Lean 

CMMI®  Processes  and  Procedures 
Using  Architectures  and  Models 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

1B2  -  Tutorial  (Continued) 

13304  -  Making  Process 

Improvement  Work  -  Tying 
Improvement  and  CMMI®  Directly  to 
What  You  Care  About 

Mr.  Neil  Potter,  The  Process  Group 

1B3  -  Tutorial  (Continued) 

13492  -  Lesson  Learned  from  Pilot 
Implementation  of  Organizational 
Performance  Management  (0PM) 
Process  Area 

Mr.  Kobi  Vider,  K.  VP.  Consulting 

1B4  -  Tutorial  (Continued) 

13431  -  Risk  and  Issue 

Management 

Mr.  Al  Florence,  MITRE 

Lunch  in  Grand  Mesa  ABC,  2nd  Floor 

1C1  -  Tutorial 

1 C2  -  Tutorial 

1 03  -  Tutorial 

1 04  -  Tutorial 

1 :00  pm 

2:45  pm 

1 3437-  How  to  Design  the  ‘Vital 

Few’  Lean  Metrics 

1 3555  -  Architecture:  Why  Your 
CMMI®  VI  .3  Implementation  is 
Incomplete  Without  It! 

1 3493  -  Leveraging  Your  Service 
Quality  Using  ITIL  V3,  ISO  20000  and 
CMMI®-SVC 

1 3479  -  Software  Safety 

Session  C 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

Dr  Lawrence  Jones  &  Dr.  Michael 
Konrad,  Software  Engineering 

Institute 

Mr.  Kobi  Vider,  K.  V.P  Consulting 

Mr.  Tim  Kasse,  National  Renewable 
Energy  Laboratory  (NREL) 

Break  in  Grand  Mesa  Foyer,  2nd  Floor 

1D1  -  Tutorial  (Continued) 

1D2  -  Tutorial  (Continued) 

1D3  -  Tutorial  (Continued) 

1D4  -  Tutorial  (Continued) 

3:15  pm 

5:00  pm 

1 3437-  How  to  Design  the  ‘Vital 

Few’  Lean  Metrics 

1 3555  -  Architecture:  Why  Your 
CMMI®  VI  .3  Implementation  is 
Incomplete  Without  It! 

1 3493  -  Leveraging  Your  Service 
Quality  Using  ITILV3,  ISO  20000  and 
CMMI®-SVC 

1 3479  -  Software  Safety 

Session  D 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

Dr  Lawrence  Jones  &  Dr.  Michael 
Konrad,  Software  Engineering 

Institute 

Mr.  Kobi  Vider,  K.  V.P.  Consulting 

Mr.  Tim  Kasse,  National  Renewable 
Energy  Laboratory  (NREL) 

**Additional  Cost  Required** 
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TUESDAY,  NOVEMBER  15, 2011 


Track  1 

Track  2 

Track  3 

Track  4 

Mesa  Verde 

Wind  Star 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Services 

Agile/Lean 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Mr.  Steve  Henry,  Northrop 
Grumman  Corporation 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

1 :30  pm 

1 3488  -  Just  Getting  Started  with 
CMMI® 

1 3427  -  Why  Do  Developers  Make 
These  Dangerous  Software  Errors? 

12936  -Applying  CMM®-SVC 
Process  Areas  as  an  Extension  to 
CMM®-DEV 

1 3288  -  Comparing  Scrum  And 
CMMI®  -  How  Can  They  Work 
Together? 

2:15  pm 

Ms.  Mary  Beth  Chrissis,  Software 
Engineering  Institute 

Mrs.  Michele  Moss,  Booz  Allen 
Hamilton 

Ms.  Mary  Lynn  Penn,  Lockheed 

Martin  IS&GS 

Mr.  Neil  Potter,  The  Process  Group 

2:15  pm 

1 3481  -  Are  Your  Customers  Happy? 
Using  Customer  Satisfaction  to  Drive 
Improvement  Efforts 

1 3483  -  Your  Destiny,  It’s  In  Your 
Defect  Density 

1 3263  -  A  LEAN  and  RACI  Approach 
to  CMMI®  for  Services  (CMMI®- 
SVC) 

13439  -  Using  a  Lean  Measurement 
Framework  to  Design  CMMI® 

Metrics 

3:00  pm 

Ms.  Susan  LaFortune,  Department 
of  Defense 

Ms.  Margaret  Tanner  Glover,  MITRE 

Mr.  Ahn  Nuzen,  SPAWAR  Systems 
Center  Pacific  (SSC  PAC) 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Services 

Agile/Lean 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Mr.  Steve  Henry,  Northrop 
Grumman  Corporation 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

3:30  pm 

13510  -  Lessons  Learned  in 
Overcoming  Resistance  to  CMMI 

1 3429  -  NAVSEA  Efforts  to  Bring 
Reliability  and  Maintainability 
Engineering  into  Software  Reliability 

1 3291  -  A  Real-Life  Example  of 
Appraising  and  Interpreting  CMMI®- 
Services  Maturity  Level  2 

1 3446  -  Utilizing  Lean  Six  Sigma  to 
Attain  CMMI®  Maturities 

4:15  pm 

Dr  Rick  Hefner,  Northrop  Grumman 
Corporation 

Mr.  Paul  Dube,  NAVSEA 

Mr.  Neil  Potter,  The  Process  Group 

Mr.  Steven  Moffat,  SAIC 

4:15  pm 

5:00  pm 

13456  -  Diagnosis  PIF:  Avoiding 
Process  Improvement  Fatigue 

Mr.  Craig  Hale,  Esterline  Control 
Systems  -AVISTA 

13365  -  Some  Assembly  Required: 
Using  Agile  Methodologies  to 

Develop  an  Interactive  Software 

User’s  Guide 

Mr.  Ronald  Stauffer,  Raytheon 
Company 

13520  -CMMI®-DEV:  Where  “Build 
Stuff”  Happens  in  CMMI®-SVC 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

1 3465  -  Design  Your  Business 
Processes  to  Embrace  People  in  an 
Agile  Approach  and  Support  High 
Maturity  (0PM) 

Mr.  Kobi  Vider,  K.  1 IP.  Consulting 

CMMI®  TECHNOLOGY  CONFERENCE 
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WEDNESDAY,  NOVEMBER  16, 2011 


Track  1 

Track  2 

Track  3 

Track  4 

Mesa  Verde 

Wind  Star 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

High  Maturity 

Appraisals 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Mr.  Steve  Henry,  Northrop 
Grumman  Corporation 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

8:30  am 

9:15  am 

1 3487  -  Ask  an  Author:  Questions 
about  CMMI®  vl.3 

Ms.  Mary  Beth  Chrissis,  Software 
Engineering  Institute 

1 331 8  -  Chutes  and  Ladders  - 
CMMI®  -  ISO  Considerations 

Ms.  Kimberley  Eley,  ManTech 

13454  -  Flardware  High  Maturity  - 
It’s  Not  Software  or  Defects 

Mr.  Tom  Lienhard,  Raytheon 

1 3290  -  Appraisals  and  CMMI® 
Gotchas  -  Lessons  in  CMMI®  Use 
and  Appraisal  Preparation 

Mr.  Neil  Potter,  The  Process  Group 

9:15  am 

1 351 4  -  The  Fundamental  Value  of 
Every  Single  CMMI®  Practice 

1 3477  -  A  Tale  of  Two  Cultures 

1 3459  -  A  Process  Performance 
Models  Case  Study 

13451  -  Streamlining  Process 
Deployment  and  Compliance 

10:00  am 

Dr  Rick  Hefner,  Northrop  Grumman 
Corporation 

Ms.  Beth  Layman,  Layman  & 

Layman 

Mr.  Kobi  Vider,  K.  V.P  Consulting 

Mr.  Gary  Natwick,  Harris  Corporation 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

High  Maturity 

Appraisals 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Mr.  Steve  Henry,  Northrop 
Grumman  Corporation 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

10:30  am 

11:15  am 

1 3475  -  Questions  from  the 

Trenches:  What  Over  1 000  Students 
Want  to  Know  Most  About  the 

CMMI® 

Mr.  Bill  Smith,  Leading  Edge  Process 
Consultants 

1 3303  -  Using  Lessons  Learned 
from  Medical  Checklists  to  Simplify 
CMMI®  Processes 

Mr.  Neil  Potter,  The  Process  Group 

1 3500  -  Use  Causal  Analysis  and 
Resolution  (CAR)  &  Quantitative 

Project  Management  (QPM)  to 

Monitor  and  Control  the  Quality  and 
Process-Performance 

Mr.  YasirMadi,  OST  Global 

1 351 2  -  Flow  to  Successfully  and 
Cost-Effectively  Conduct  a  Re- 
Appraisal 

Dr.  Rick  Hefner,  Northrop  Grumman 
Corporation 

11:15  am 

12:00  pm 

13450  -  CMMI®  Training  -  Avoiding 
Waste  and  Ineffectiveness 

Ms.  Sari  Schneider,  The  Boeing 
Company 

13480  -  Creating  an  SQA  Program 
at  the  National  Renewable  Energy 
Laboratory  (NREL)  -  A  DOE  FFRDC 
Based  on  the  NDIA/DoD  Sponsored 
CMMI® 

Mr.  Tim  Kasse,  National  Renewable 
Energy  Laboratory 

1 3508  -  Strategies  for  Retaining 
CMMI®  Maturity  Level  5  in  vl  .3 

Dr  Rick  Hefner,  Northrop  Grumman 
Corporation 

13240  -  CMMI®  Maturity  Level  5 
—  A  Bargain! 

Mrs.  Jayne  Perkins,  Raytheon  IDS 

CMMI®  TECHNOLOGY  CONFERENCE 
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WEDNESDAY  (CONTINUED) 


Track  1 

Track  2 

Track  3 

Track  4 

Mesa  Verde 

Wind  Star 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Multi-Model  Improvement 

Appraisals 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

1:30  pm 

2:15  pm 

13498  -  Communication  Loops  in 
CMMI®  v.1.3 

Mr.  Shawn  Presson,  TestPros,  Inc. 

1 3496  -  TSP  and  Architecture  in  the 
Real  World 

Mr.  James  McHale,  Software 
Engineering  Institute 

13438  -  Using  the  U.S.  Malcolm 
Baldrige  National  Quality  Award 
Performance  Criteria  to  Measurably 
Improve  CMMI®  Results 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

13402  -  CMMI®  Surveillance 
Appraisals:  A  Modest  Proposal 

NDIA  CMMI®Working  Group 

2:15  pm 

3:00  pm 

1 3472  -  Why  Project  Managers 
(Understandably)  Hate  the  CMMI® 

—  and  What  to  Do  About  It 

Mr.  Bill  Smith,  Leading  Edge  Process 
Consultants 

13433  -  Risk  Management 

Mr.  Al  Florence,  The  MITRE 
Corporation 

13516  -  Assembling  a  Multi- 
Model  Approach  to  Improving 

Service  Quality  and  Ensuring 

Service  Resilience  in  Complex  Risk 
Environments 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

1 2987  -  SCAMPI  Planning  with  vl  .3 

Ms.  Dorna  Witkowski,  Lockheed 

Martin  Corporation 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Multi-Model  Improvement 

Appraisals 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

3:30  pm 

4:15  pm 

13495  -CMMI®  Level  1  to  3  in 

1 5  Months:  AIM  for  a  Performance 
Upgrade 

13511  -  Effective,  CMMI-Compliant 
Project  Plans  (in  Less  Than  1 0 

Pages) 

12935  -  CMMI®  for  Acquisition 
(CMMI®-ACQ)/CMMI®  for  Services 
(CMMI®-ACQ)  and  Their  Potential 
Contribution  on  the  Rapid  Acquisition 
of  IT  Systems  in  Support  of  Public 

Law  111 

13506  -  Increasing  Efficiency  and 
Saving  Costs  with  New  SCAMPI 
Approaches 

Dr  Gene  Miluk,  Software  Engineering 
Institute,  Carnegie  Mellon  University 

Dr  Rick  Hefner,  Northrop  Grumman 
Corporation 

Dr  Kenneth  Nidiffer,  Software 
Engineering  Institute 

Mr.  Paul  Byrnes,  ISD,  Inc. 

4:15  pm 

5:00  pm 

1 3458  -  Project  Driven  Process 
Improvement:  A  large  Scale  Lean  Six 
Sigma  Deployment  in  Egypt  Across 
Multiple  Small  and  Medium  Sized 
Software  Organizations 

Dr  Radouane  Oudrhiri,  Systonomy 

13443  -  Using  Lean  Architectures 
and  Models  to  Achieve  Measurable 
Reuse 

Mr.  Tim  Olson,  Lean  Solutions 

Institute,  Inc. 

13452  -  Implementation  of  Process 
Improvement  in  a  Multi-Model/ 
Standard  Government  Environment 

Mr.  Nathaniel  Becker,  U.S.  Army 
ARDEC 

13485  -  Experiences  in  the 

Applying  of  MDD  (Method  Definition 
Document)  vl  .3  Sampling  Factors 
and  Sampling  Algorithm 

Mr.  Bradley  Bittorf,  Raytheon 

Company 
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AND  USER  GROUP  ►  8 


THURSDAY,  NOVEMBER  17, 2011 


Track  1 

Track  2 

Track  3 

Track  4 

Mesa  Verde 

Wind  Star 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Multi-Model  Improvement 

Appraisals 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

8:30  am 

1 3644  -  CMMI  for  Executives 

1 3028  -  Process  Acceptance 

Through  Use  Cases 

13491  -  CMMI®,  ISO  and  AS9100, 

An  Efficient  and  Effective  Approach 

1 3449  -CMMI®  Without  Appraisals 

9:15  am 

NDIA  CMMI®Working  Group 

Dr  Juergen  Schmied,  Method  Park 

Mrs.  LaKeisha  Souter,  Northrop 
Grumman  Corporation 

Mr.  Alan  Gellis,  The  Boeing  Company 

9:15  am 

10:00  am 

1 3471  -  Cracking  the  Code  for 
Improving  the  Productivity  of 
Knowledge  Workers 

Mr.  Peter  Voldby  Petersen,  Callis 

1 3482  -  How  a  Focus  on  High 

Maturity  CMM®-Based  Process 
Improvement  Can  Add  Value  to  the 
Organizations  Even  When  There  are 
Only  Limited  Resources 

Mr.  Edmond  Sung,  Processis,  LTD 

13468  -  Using  the  SEI  Models 
and  Practices  to  Assure  your 
Subcontractors  quality  with  Cross 
Constellations  and  Multi-Models 
Inspiration 

Mr.  Kobi  Vider,  K.  V.P  Consulting 

1 3643  -  SCAMPI  A  VI  .3  MDD  Usage 
and  Profile 

Mr.  Michael  Campo,  Raytheon 
Company 

Break  in  Grand  Mesa  Foyer,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Services 

Agile/Lean 

Session  Chairs: 

Mr.  Mike  Campo,  Raytheon 
Company;  Ms.  Mary  Beth 
Chrissis,  Software  Engineering 
Institute 

Session  Chairs: 

Ms.  Juliet  Davis,  The  Boeing 
Company;  Mr.  Geoff  Draper, 
Harris  Corporation 

Session  Chair: 

Ms.  Eileen  Forrester,  Software 
Engineering  Institute 

Session  Chairs: 

Dr.  Mike  Konrad,  Software 
Engineering  Institute;  Ms.  Beth 
Layman,  Layman  and  Layman, 

Inc. 

10:30  am 

11:15  am 

13455  -  Improving  Process  Quality 
While  Lowering  Costs 

Mr.  Craig  Hale,  Esterline  Control 
Systems  -  AVISTA 

12962  -  Parametric  Estimation  for 

ERP  Implementations 

Mr.  Donald  Beckett,  Quantitative 
Software  Management 

1 3460  -  Lean  Six  Sigma,  Towards 
an  Empirical  and  Experimental 
Approach  to  Software  Process 

Dr  Radouane  Oudrhiri,  Systonomy 

13421  -  You  are  a  CMMI®-DEV 
Appraisal  Expert?  What  Do  You  Do 

If  Your  Organization  Wants  to  Do  a 
CMMI®-SVC  Appraisal? 

Mr.  Joseph  Trujillo,  Raytheon 

11:15  am 

12:00  pm 

1 3505  -  The  Change  Comes  from 
the  Inside!  Divide  and  Conquer:  A 

Top  Down  and  Bottom  Up  Approach 

Mr.  Surya  Kanchiraju,  OST,  Inc. 

13434  -  Writing  Requirements 
Properly 

Mr.  Al  Florence,  The  MITRE 
Corporation 

1 341 9  -  Raytheon  Pasadena 
Operations,  CMM®-DEV  versus 
CMM®-SVC  Analysis 

Mrs.  Rose-Marie  Gonzalez,  Raytheon 

1 3404  -  Managed  Discovery? 

PIIDs?  How  Do  1  Decide  Which 

SCAMPI  Data  Collection  Techniques 
are  Right  for  My  Organization 

Mr.  Sam  Fogle,  ACE  Guides 

CMMI®  TECHNOLOGY  CONFERENCE 
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ADDITIONAL  AUTHORS 


ABSTRACT# 

ABSTRACT  TITLE 

ADDITIONAL  AUTHORS 

13240 

CMMI  Maturity  Level  5 . A  Bargain! 

Mrs.  Mary  Ellen  Christopher;  Mr. 

Ralph  Williams 

13263 

A  LEAN  and  RACI  Approach  to  CMMI  for  Services  (CMMI-SVC) 

Mr.  David  A.  Dayton 

13318 

Chutes  and  Ladders  -  CMMI-ISO  Considerations 

Ms.  Kate  Roehling 

13402 

CMMI  Surveillance  Appraisals:  A  Modest  Proposal 

Mr.  Michael  Campo 

13403 

CMMI:  Back  to  the  Future? 

Mr.  Michael  Campo 

13419 

Raytheon  Pasadena  Operations,  CMMI-DEV  versus  CMMI-SVC  Analysis 

Mr.  G.  Doug  Mazezka 

13421 

You  are  a  CMMI-DEV  appraisal  expert?  What  do  you  do  if  you  org  wants  to  do  a 
CMMI-SVC  appraisal? 

Ms.  Debra  L.  Smith 

13446 

Utilizing  Lean  Six  Sigma  to  attain  CMMI  maturities 

Mr.  Ahn  T.  Nuzen 

13458 

Project  Driven  Process  Improvement  -  A  large  scale  Lean  Six  Sigma  deployment  in 
Egypt  across  multiple  small  and  medium  sized  software  organisations 

Dr.  Amr  Kamel 

13477 

A  Tale  of  Two  Cultures 

Ms.  Marie  Johnson 

13481 

Are  Your  Customers  Happy?  Using  Customer  Satisfaction  to  Drive  Improvement  Efforts 

Ms.  Kathleen  A.  Demery 

13482 

How  a  focus  on  high  maturity  CMMI-based  process  improvement  can  add  value  to  the 
organization  even  when  there  are  only  limited  resources 

Mr.  Tim  Kasse 

13491 

CMMI,  ISO,  and  AS91 00,  an  Efficient  and  Effective  Approach 

Mr.  Albert  1.  Chatmon 

13495 

CMMI  Level  1  to  3  in  1 5  Months:  AIM  for  a  performance  upgrade 

Mr.  Timothy  A.  Chick 

13500 

Use  Causal  Analysis  and  Resolution  (CAR)  &  Quantitative  Project  Management  (QPM) 
to  monitor  and  control  the  quality  and  process-performance 

Mrs.  Kusum  Dhyani 

13505 

The  change  comes  from  the  inside!  Divide  and  Conquer:  A  Top  Down  and  Bottom  Up 
Approach 

Ms.  Carolina  Rivero 

13506 

Increasing  efficiency  and  saving  costs  with  new  SCAMPI  approaches 

Mr.  Jeff  McGarry 

13511 

Effective,  CMMI-Compliant  Project  Plans  (in  Less  Than  1 0  Pages) 

Mr.  Ferol  Lewis 

13555 

Tutorial  -  Architecture:  Why  your  CMMI  VI  .3  implementation  is  incomplete  without  it! 

DR.  Michael  Konrad 
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NATIONAL  DEFE 
ASSOCIATION 

2111  WILSON  B( 
ARLINGTON,  VA: 
(703)  522-1820 
(703)  522-1885 
WWW.NDIA.ORG 


CMMI®| 
TECHNI 
CONFEI 
AND  U! 
GROUP 

Investigate 
and  Lessor 
About  the 
Between 
Capability 
Program  PJ 

NOVEMBEf 

HYATT  REG| 
TECH  CEN1 

DENVER,  Cl 


TO  REGISTE 

WWW.NDIA.ORG/ME 


NDIK 

National  Defense  Industrial  Association 


What  Do  You  Know  About  Process  Improvement 

and  Physics-Based  Modeling? 

CMMI  Technology  Conference  and  User  Group 

Physics-based  Modeling  in  Design  &  Development  for 

U.S.  Defense  Conference 

November  16,  201 1 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


1 


Welcome! 


Objective: 

-  Help  build  awareness  and  interest  in 
process  improvement  and  physics-based 
modeling 

-  Facilitate  integration  between  communities 

Approach: 

-  Brief  overview  of  fundamental  concepts 

-  Interactive  responses  to  targeted  questions 
for  community  feedback 


Note:  Please  return  the  interactive  voting  devices. 
They  are  on  loan  courtesy  of  Harris  Corporation. 


NPIK 

National  Defense  Industrial  Association 


11th  ANNUAL 
CMMI®  TECHNOLOGY 
CONFERENCE  AND 
USER  GROUP 


PHYSICS-BASED 
MODELING  IN  DESIGN  & 
DEVELOPMENT 
FOR  U.S.  DEFENSE 

CONFERENCE  D»D  Acquisition" 


s  -»3r  NOVEM  BE  R  1 4- 1 7.  201 1 

CCmitlTt  WWfl.lttlXBRGWffTIHGS/7170 


NDIA  Systems  Engineering  Division 

11/16/2011 
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I  am  here  primarily  for... 


MPIK 

National  Defense  Industrial  Association 


1.  CMM I  Conference 

2.  Physics-Based  Modeling 
Conference 

3.  Both  conferences 

4.  The  food  and  drinks 


Oof  5 


25%  25%  25%  25% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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What  domain  are  you  representing? 


NPIK 

National  Defense  Industrial  Association 


1 .  Defense  industry 

2.  Commercial  industry 

3.  Government 

4.  Academia 

5.  FFRDC 

6.  Other 


Oof  5 


0%  0%  0%  0%  0%  0% 

/  ^  <=>  <=>  <z=>  \ 


NDIA  Systems  Engineering  Division 

11/16/2011 
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How  large  is  your  organizational  unit?  MDIK 

National  Defense  Industrial  Association 

1 .  <50  people 

2.  50-250  people 

3.  250-1,000  people 

4.  1,000-5,000  people 

5.  5,000-10,000  people 

6.  >  10,000  people 


0%  0%  0%  0%  0%  0% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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What  Is  CMMI? 


NDIK 


National  Defense  Industrial  Association 


CMMI  is  a  model  representing  a  collection  of  best  practices  proven 
effective  in  industry 

•  A  framework  for  developing,  improving,  and  sustaining  business  performance 

•  Provides  a  process  focus  on  work  activities 

•  Developed  by  industry  (commercial  and  defense),  government,  academia 

CMMI  targets  three  primary  environments: 

•  Development  - - 


Engineering  a  product  or  service 


CMMI-DEV 


•  Services  - 

Providing  services 


•  Acquisition  - 

Acquiring  products  and  services 


The  CMMI  product  suite 
consists  of: 

•  Models  and  primers 

•  Appraisal  methods 

•  Training  courses 


Source:  CMMI®  for  Executives,  NDIA  Systems  Engineering  Division. 
http://www.ndia.orq/Divisions/Divisions/SvstemsEnqineerinq/ 


Capability  Maturity  Model  Integration  (CMMI®) 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Does  your  organization  have  a  CMMI 
maturity  level  rating? 

1 .  ML  1 

2.  ML  2 

3.  ML  3 

4.  ML  4 

5.  ML  5 

6.  No  rating 


NPIK 

National  Defense  Industrial  Association 


Oof  5 


0%  0%  0%  0%  0%  0% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Which  model  constellations  has  your  organization  used, 
or  will  use  in  the  next  year?  (select  all  that  apply) 


1.  CMMI-DEV 

2.  CMMI-SVC 

3.  CMMI-ACQ 

4.  People  CMM 

5.  None 


Oof  5 


MPIK 

National  Defense  Industrial  Association 


20%  20%  20%  20%  20% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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How  does  your  organization  implement 
process  improvement? 


NDIK 

National  Defense  Industrial  Association 


1 .  None,  or  not  applicable 

2.  Primarily  software 

3.  Primarily  engineering 

4.  Separate  functional  process 
groups 

5.  Coordinated  cross- 
functionally  (Engineering, 

PM,  Mfg,  Quality,  etc.) 

0%  0%  0%  0%  0% 


Oof  5 


NDIA  Systems  Engineering  Division 
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What  top  organizational  improvement  goals  drive  your 
measurable  improvement  initiatives? 

(select  up  to  3) 


NDIK 

National  Defense  Industrial  Association 


1 .  Reduce  cost 

2.  Cycle  time 

3.  Improve  product  quality  (reduce 
rework) 

4.  Operational  effectiveness 

5.  Customer  satisfaction 

6.  Workforce  capability/skills 

7.  Expanding  technologies 
(e.g.,  Agile) 

8.  Penetration  into  new  markets 

9.  Other 


Oof  5 


11%  11%  11%  11%  11%  11%  11%  11%  11% 


NDIA  Systems  Engineering  Division 
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How  do  you  measure  the  effectiveness  of  your 
improvement  initiatives?  (select  up  to  3  choices) 


MPIK 

National  Defense  Industrial  Association 


1 .  Qualitative  (not  measured) 

2.  Basic  measures  for  individual 
initiatives 

3.  Organizational  improvement 
trends  (e.g.,  across  projects) 

4.  Return  on  Investment  (ROI) 

5.  Statistical  process  control  (SPC) 

6.  Lean/Six  Sigma 

7.  Other 


14%  14%  14%  14%  14%  14%  14% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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What  real  impact  has  process  improvement 
had  on  your  business  performance? 

1.  High  value 

2.  Moderate  value 

3.  Low  value 

4.  None 

5.  Negative 


NPIK 

National  Defense  Industrial  Association 


Oof  5 


0%  0%  0%  0%  0% 

X  <^>  <CZ>  CZTa  \ 
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Measures  of  CMMI  Adoption  and 
Performance  Benefits 


MPIK 

National  Defense  Industrial  Association 


•  Studies 


•  Reports 

•  Workshops 

•  Analyses 


A  Srftwwc  Lnoinotmitg  Institute 

Use  Lind  Organizational  Effects  of  Measurement 
llikI  Analysis  m  Hij»li  Mutitriix  Organisations: 
Resulls  from  llie  2lM)8  SE1  Stale  of  MeLisuremenl 
and  Analysis  Practice  Surveys 


http://www.sei.cmu.edu/librarv/assets/ 

presentations/CMMI%20Benefits.pdf 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


May  2010 


r  Software  Engineering  Institute  Carnegie  Mellon 


http://www.sei.cmu.edu/librarv/ 

abstracts/reports/08tr024.cfm 


http://www.sei.cmu.edu/cmmi/ 

casestudies/profiles/pdfs/ 

upload/201 1SepCMMI.pdf 


CMMI"  for 

SCAMPISM  Class  A  Appraisal  Results 
2011  Mid-Year  Update 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh.  PA  15213 

September  201 1 


4^-  Software  Engineering  Institute  Carnegie  Met  Ion 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Industry  Benefits  from  CMMI 


NPIK 

National  Defense  Industrial  Association 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie-  Mellon  Univefsity 
Pittsburgh,  PA  15£13 

May  2010 


So  ft  warn  Engineering  Institute  (jirncpcMi'lljHi 


Example  measures  reported  by  NDIA  member  companies: 

Defect  repair  effort 

•Defect  repair  hours:  -58%  (ML3  to  ML5) 
•Defect  cost  savings:  -105  hrs  per  defect 
•l&T  hrs/defect:  -24% 

•Hours/KLOC:  -22%  (ML3  to  ML5) 

Defect  density 

•62%  fewer  high-severity  defects  (ML5) 
•Defect  phase  containment:  +240%  (ML5) 
•>85%  defects  removed  prior  to  sys  test 
•Acceptance  test:  <  0.15  defects/KLOC 

Development  cost 

•SW  development  cost:  -28%  (ML3  to  ML5) 
•Potential  project  savings:  $1 .9M-$2.3M 

Productivity 

•Productivity  gain:  +42%  (ML5,  9yrs) 

Cost/schedule 

•Over  6X  less  likely  cost/schedule  impact 

The  new  data  presented  in  this  report  demonstrates  that  effective 
implementation  of  good  practices  aided  by  use  of  CMMI  can  improve 
cost,  schedule,  and  quality  performance. 


Benefits  of  CMMI  in  the  Defense  Industry,  Software  Engineering  Institute,  May  2010. 
http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense-lndustrv.cfm 

Source:  CMMI®  for  Executives,  NDIA  Systems  Engineering  Division. 
http://www.ndia.org/Divisions/Divisions/SvstemsEnqineerinq/ 
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What  is  Physics-Based  Modeling?  MPIK 

^  National  Defense  Industrial  Association 


F-18 


■  P'WwlL 


Physics-Based  Modeling  is  the  use  of  computer  software 
applications  to  design  and  analyze  the  performance  of 
systems  by  solving  the  physics  equations  that  govern  the 
behavior  of  the  system 

Reduces  the  need  for  physical  testing 
Examples: 

-  Air  Vehicles — predict  air  vehicle  performance 
by  solving  the  equations  that  describe  the 
airflow  over  the  wings  and  other  components  vortex  induced  Tail  Buffet 
(i.e.  lift  and  drag)  together  with  the 
deformation  of  the  air  vehicle  structure  due  to 
the  airflow  loads. 

-  Tires — predict  tire  performance  by  solving  the 
equations  that  describe  tire  flexing, 
deformation,  and  treadweare.g.  behavior  of 
rubber,  steel  belts,  water  flow  in  treads,  etc. 


~  60  Million  Cycles 
During  an  80,000  Mile 
Tire  Lifetime 


NDIA  Systems  Engineering  Division 

11/16/2011 
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CREATE  Program 


NPIK 

National  Defense  Industrial  Association 


The  Computational  Research  &  Engineering  Acquisition  Tools  and 
Environments  (CREATE)  project  was  initiated  by  the  Department  of  Defense 
in  2008  as  a  long  term  effort  to  enable  major  improvements  in  the  acquisition 
process  with  the  following  goals: 

■Prevent  defects  and  design  flaws  early  in  the  acquisition  process 
■Reduce  rework  thereby  enabling  faster  system  deployment 
■Reduce  experimental  testing  time  and  effort  through  analysis  of  virtual 
prototypes 

It  would  accomplish  this  by  injecting  multi-physics  based  predictions  early 
within  the  design  and  analysis  process,  by  developing  and  deploying 
production  quality  design  and  analysis  software  that  is  adaptable  and 
maintainable,  and  developing  and  deploying  multi-physics  based 
Computational  Engineering  tools  that  exploit  next  generation  high-speed 
computer  resources.  These  tools  have  the  real  potential  to  dramatically  reduce 
the  overall  time  and  effort  required  to  design  and  develop  DoD  systems  and 
products  in  fixed  and  rotary  wing  aircraft,  ships,  propulsion  systems,  antennae, 
and  related  systems,  while  greatly  improving  accuracy  as  well. 


NDIA  Systems  Engineering  Division 

11/16/2011 
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CREATE  Products 


NPIK 

National  Defense  Industrial  Association 


Air  Vehicles 

DaVinci  -  Rapid  conceptual  design  development  and  initial  system  integration 
Kestrel  -  High-fidelity,  full  vehicle,  multi-physics  analysis  tool  for  fixed-wing  aircraft- 
Release  1 .0  Provides  Rigid  body  CFD  fixed  wing  AV  with  preliminary  aeroelastics 
Helios  -  High-fidelity,  full  vehicle,  multi-physics  analysis  tool  for  rotary-wing  aircraft 
-  Release  1 .0  Provides  highly  accurate  calculation  of  rotorcraft  vortex  shedding 
Firebolt  -  Module  for  propulsion  systems  in  fixed  and  rotary-wing  air  vehicles 
Ships 

RDI  -  Rapid  Design  and  Synthesis  Capability — Partnership  with  ONR  and  NAVSEA 
NESM  -  Ship  Shock  &  Damage-prediction  of  shock  and  damage  effects  -  Release 
0.1  provides  initial  ship  shock  vulnerability  analysis  for  underwater  explosions 
NAVYFOAM  -  Ship  Hydrodynamics-predict  hydrodynamic  performance 
IHDE  -  Environment  to  facilitate  access  to  Naval  design  tools  -  Release  1 .0 
provides  initial  user  interface  for  ship  hydrodynamics 
RF  Antenna 

SENTRI  -  Electromagnetics  antenna  design  integrated  with  platforms  -  Release  1.0 
and  1.5  provides  Initial  RF  antenna  design  and  analysis  with  V&V 

Meshing  and  Geometry 

Capstone  -  Components  for  generating  geometries  and  meshes 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


18 


Does  your  organization  now  use  any  physics-based 
modeling  as  part  of  your  design  environment? 


MPIK 

National  Defense  Industrial  Association 


1 .  Not  used  currently 

2.  Some  use  currently 

3.  Significant  use  currently 

4.  Don’t  know 


25%  25%  25%  25% 

n  n  n  n 


10 


Oof  5 


NDIA  Systems  Engineering  Division 
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What  is  your  design  domain? 
(select  all  that  apply) 

1.  Ships 

2.  Fixed-wing  aircraft 

3.  Missiles 

4.  Helicopters 

5.  Radar 


Oof  5 


MPIK 

National  Defense  Industrial  Association 


20%  20%  20%  20%  20% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Thank  you  for  your  participation!  NDIK 

National  Defense  Industrial  Association 

•  To  learn  more  about  CMMI  or  Physics- 
Based  Modeling  contact  the  NDIA  SE 
Division  chair 

-  Bob  Rassa  (rcrassa@ravtheon.com) 


Please  return  the  interactive  voting  devices! 


NDIA  Systems  Engineering  Division 
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NDIK 

National  Defense  Industrial  Association 


What  Do  You  Know  About  Process  Improvement 

and  Physics-Based  Modeling? 

CMMI®  Technology  Conference  and  User  Group 

Physics-based  Modeling  in  Design  &  Development  for 

U.S.  Defense  Conference 

November  16,  201 1 


®  CMMI  is  a  registered  trademark  of  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Welcome! 


Objective: 

-  Help  build  awareness  and  interest  in 
process  improvement  and  physics-based 
modeling 

-  Facilitate  integration  between  communities 

Approach: 

-  Brief  overview  of  fundamental  concepts 

-  Interactive  responses  to  targeted  questions 
for  community  feedback 


Note:  Please  return  the  interactive  voting  devices. 
They  are  on  loan  courtesy  of  Harris  Corporation. 


NPIK 

National  Defense  Industrial  Association 


11th  ANNUAL 
CMMI®  TECHNOLOGY 
CONFERENCE  AND 
USER  GROUP 


PHYSICS-BASED 
MODELING  IN  DESIGN  & 
DEVELOPMENT 
FOR  U.S.  DEFENSE 

CONFERENCE  D»D  Acquisition" 


s  -»3r  NOVEM  BE  R  1 4- 1 7.  201 1 

CCmitlTt  WWfl.lttlXBRGWffTIHGS/7170 
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What’s  the  Relationship  between  CMMI 
and  Physics-Based  Modeling? 


NDIK 

National  Defense  Industrial  Association 


Process  integrates 
technology  with  resources 

•  Technology,  by  itself,  will 
most  likely  not  be  used 
effectively 


PEOPLE 


NDIK 


11™  ANNUAL 
CMMI®  TECHNOLOGY 
CONFERENCE  AND 
USER  GROUP 


PHYSICS-BASED 
MODELING  IN  DESIGN  & 
DEVELOPMENT 
FOR  U.S.  DEFENSE 
CONFERENCE 


CMMI  Technology  Conference 
and  User  Group 


Physics-Based  Modeling  in  Design  & 
Development  for  U.S.  Defense  Conference 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


3 


I  am  here  primarily  for... 


NPIK 

National  Defense  Industrial  Association 


1.  CMM I  Conference 

2.  Physics-Based  Modeling 
Conference 

3.  Both  conferences 

4.  The  food  and  drinks 


47%  47% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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What  domain  are  you  representing?  MDIK 

National  Defense  Industrial  Association 


1 .  Defense  industry 

2.  Commercial  industry 

3.  Government 

4.  Academia 

5.  FFRDC 

6.  Other 


47% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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How  large  is  your  organizational  unit?  MDIK 

National  Defense  Industrial  Association 


1 .  <50  people 

2.  50-250  people 

3.  250-1,000  people 

4.  1,000-5,000  people 

5.  5,000-10,000  people 

6.  >  10,000  people 


23% 


NDIA  Systems  Engineering  Division 

11/16/2011 
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What  Is  CMMI? 


NDIK 


National  Defense  Industrial  Association 


CMMI  is  a  model  representing  a  collection  of  best  practices  proven 
effective  in  industry 

•  A  framework  for  developing,  improving,  and  sustaining  business  performance 

•  Provides  a  process  focus  on  work  activities 

•  Developed  by  industry  (commercial  and  defense),  government,  academia 

CMMI  targets  three  primary  environments: 

•  Development  - - 


Engineering  a  product  or  service 


CMMI-DEV 


•  Services  - 

Providing  services 


•  Acquisition  - 

Acquiring  products  and  services 


The  CMMI  product  suite 
consists  of: 

•  Models  and  primers 

•  Appraisal  methods 

•  Training  courses 


Source:  CMMI®  for  Executives,  NDIA  Systems  Engineering  Division. 
http://www.ndia.orq/Divisions/Divisions/SvstemsEnqineerinq/ 


Capability  Maturity  Model  Integration  (CMMI®) 


NDIA  Systems  Engineering  Division 

11/16/2011 
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CMMI  Model  Structure 


MPIK 

National  Defense  Industrial  Association 


Source:  CMMI®  for  Executives,  NDIA  Systems  Engineering  Division. 
http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/ 


Benchmark  Ratings 


•Goals 

•Process  Areas 


•Maturity  Levels 
•Capability  Levels 


CMMI-DEV 


•Requirements  Development 
•Supplier  Agreement  Mgmt 
•Technical  Solution 
•Product  Integration 
•Verification 
•Validation 


CMMI-SVC 

•Capacity  &  Availability 
Management 
•Incident  Resolution  and 
Prevention 

•Supplier  Agreement  Mgmt 
•Service  Continuity 
•Service  Delivery 
•Service  System  Development 
•Service  System  Transition 
•Strategic  Service  Mgmt 

CMMI  Model  Foundation 


Incremental  Frameworks  for 
Continuous  Process  Improvement 


CMMI-ACQ 

•Agreement  Management 
•Acquisition  Requirements 
Development 

•Acquisition  Technical  Mgmt 
•Acquisition  Validation 
•Acquisition  Verification 
•Solicitation  and  Supplier 
Agreement  Development 


•Requirements  Management 
•Project  Planning 
•Project  Monitoring  &  Control 
•Measurement  &  Analysis 
•Configuration  Management 
•Process  and  Product  QA 


•Policies 

•Plans 

•Resources 


•Integrated  Project  Management 
•Risk  Management 
•Decision  Analysis  &  Resolution 
•Organizational  Process  Focus 
•Organizational  Process  Definition 
•Organizational  Training 


•Quantitative  Project  Mgmt 
•Org  Process  Performance 


Institutionalization 


•Responsibilities 

•Training 

•Control  Work  Products 


•Stakeholder  Involvement 
•Monitoring  and  Control 
•Objective  Evaluation 


•Causal  Analysis  &  Resolution 
•Org  Performance  Management 


•Management  Visibility 
•Defined  Process 

•Collect  Process  Related  Experiences 


NDIA  Systems  Engineering  Division 

11/16/2011 
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Does  your  organization  have  a  CMMI 
maturity  level  rating? 


NDIK 

National  Defense  Industrial  Association 


1 .  ML  1 

2.  ML  2 

3.  ML  3 

4.  ML  4 

5.  ML  5 

6.  No  rating 


53% 


NDIA  Systems  Engineering  Division 
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Which  model  constellations  has  your  organization  used, 
or  will  use  in  the  next  year?  (select  all  that  apply) 


1.  CMMI-DEV 

2.  CMMI-SVC 

3.  CMMI-ACQ 

4.  People  CMM 

5.  None 


NPIK 

National  Defense  Industrial  Association 


50% 


36% 


25% 


8% 


3% 


4^ 


e 


ov 
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How  does  your  organization  implement 
process  improvement? 


NDIK 

National  Defense  Industrial  Association 


1 .  None,  or  not  applicable 

2.  Primarily  software 

3.  Primarily  engineering 

4.  Separate  functional  process 
groups 

5.  Coordinated  cross- 
functionally  (Engineering, 
PM,  Mfg,  Quality,  etc.) 


47% 


16% 


14% 


14% 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


11 


What  top  organizational  improvement  goals  drive  your 
measurable  improvement  initiatives?  (select  up  to  3) 


NDIK 

National  Defense  Industrial  Association 


1 .  Reduce  cost 

2.  Cycle  time 

3.  Improve  product  quality  (reduce 
rework) 

4.  Operational  effectiveness 

5.  Customer  satisfaction 

6.  Workforce  capability/skills 

7.  Expanding  technologies 
(e.g.,  Agile) 

8.  Penetration  into  new  markets 

9.  Other 


62% 


NDIA  Systems  Engineering  Division 
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How  do  you  measure  the  effectiveness  of  your 
improvement  initiatives?  (select  up  to  3  choices) 


NPIK 

National  Defense  Industrial  Association 


1 .  Qualitative  (not  measured) 

2.  Basic  measures  for  individual 
initiatives 

3.  Organizational  improvement 
trends  (e.g.,  across  projects) 

4.  Return  on  Investment  (ROI) 

5.  Statistical  process  control  (SPC) 

6.  Lean/Six  Sigma 

7.  Other 


49% 


50% 


cT 


^  ^ 
rf 

°-  ** 


V 


<2? 

4- 


keS 
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What  real  impact  has  process  improvement 
had  on  your  business  performance? 


NPIK 

National  Defense  Industrial  Association 


1.  High  value 

2.  Moderate  value 

3.  Low  value 

4.  None 

5.  Negative 


49% 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


14 


Measures  of  CMMI  Adoption  and 
Performance  Benefits 


MPIK 

National  Defense  Industrial  Association 


•  Studies 


•  Reports 

•  Workshops 

•  Analyses 


A  Srftwwc  Lnoinotmitg  Institute 

Use  Lind  Organizational  Effects  of  Measurement 
llikI  Analysis  m  Hij»li  Mutitriix  Organisations: 
Resulls  from  llie  2lM)8  SE1  Stale  of  MeLisuremenl 
and  Analysis  Practice  Surveys 


http://www.sei.cmu.edu/librarv/assets/ 

presentations/CMMI%20Benefits.pdf 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


May  2010 


r  Software  Engineering  Institute  Carnegie  Mellon 


http://www.sei.cmu.edu/librarv/ 

abstracts/reports/08tr024.cfm 


http://www.sei.cmu.edu/cmmi/ 

casestudies/profiles/pdfs/ 

upload/201 1SepCMMI.pdf 


CMMI"  for 

SCAMPISM  Class  A  Appraisal  Results 
2011  Mid-Year  Update 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh.  PA  15213 

September  201 1 


4^-  Software  Engineering  Institute  Carnegie  Met  Ion 
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Industry  Benefits  from  CMMI 


NPIK 

National  Defense  Industrial  Association 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie-  Mellon  Univefsity 
Pittsburgh,  PA  15£13 

May  2010 


So  ft  warn  Engineering  Institute  (jirncpcMi'lljHi 


Example  measures  reported  by  NDIA  member  companies: 

Defect  repair  effort 

•Defect  repair  hours:  -58%  (ML3  to  ML5) 
•Defect  cost  savings:  -105  hrs  per  defect 
•l&T  hrs/defect:  -24% 

•Hours/KLOC:  -22%  (ML3  to  ML5) 

Defect  density 

•62%  fewer  high-severity  defects  (ML5) 
•Defect  phase  containment:  +240%  (ML5) 
•>85%  defects  removed  prior  to  sys  test 
•Acceptance  test:  <  0.15  defects/KLOC 

Development  cost 

•SW  development  cost:  -28%  (ML3  to  ML5) 
•Potential  project  savings:  $1 .9M-$2.3M 

Productivity 

•Productivity  gain:  +42%  (ML5,  9yrs) 

Cost/schedule 

•Over  6X  less  likely  cost/schedule  impact 

The  new  data  presented  in  this  report  demonstrates  that  effective 
implementation  of  good  practices  aided  by  use  of  CMMI  can  improve 
cost,  schedule,  and  quality  performance. 


Benefits  of  CMMI  in  the  Defense  Industry,  Software  Engineering  Institute,  May  2010. 
http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense-lndustrv.cfm 

Source:  CMMI®  for  Executives,  NDIA  Systems  Engineering  Division. 
http://www.ndia.org/Divisions/Divisions/SvstemsEnqineerinq/ 
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What  is  Physics-Based  Modeling?  MPIK 

^  National  Defense  Industrial  Association 


F-18 


■  P'WwlL 


Physics-Based  Modeling  is  the  use  of  computer  software 
applications  to  design  and  analyze  the  performance  of 
systems  by  solving  the  physics  equations  that  govern  the 
behavior  of  the  system 

Reduces  the  need  for  physical  testing 
Examples: 

-  Air  Vehicles — predict  air  vehicle  performance 
by  solving  the  equations  that  describe  the 
airflow  over  the  wings  and  other  components  vortex  induced  Tail  Buffet 
(i.e.  lift  and  drag)  together  with  the 
deformation  of  the  air  vehicle  structure  due  to 
the  airflow  loads. 

-  Tires — predict  tire  performance  by  solving  the 
equations  that  describe  tire  flexing, 
deformation,  and  treadweare.g.  behavior  of 
rubber,  steel  belts,  water  flow  in  treads,  etc. 


~  60  Million  Cycles 
During  an  80,000  Mile 
Tire  Lifetime 
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CREATE  Program 


NPIK 

National  Defense  Industrial  Association 


The  Computational  Research  &  Engineering  Acquisition  Tools  and 
Environments  (CREATE)  project  was  initiated  by  the  Department  of  Defense 
in  2008  as  a  long  term  effort  to  enable  major  improvements  in  the  acquisition 
process  with  the  following  goals: 

■Prevent  defects  and  design  flaws  early  in  the  acquisition  process 
■Reduce  rework  thereby  enabling  faster  system  deployment 
■Reduce  experimental  testing  time  and  effort  through  analysis  of  virtual 
prototypes 

It  would  accomplish  this  by  injecting  multi-physics  based  predictions  early 
within  the  design  and  analysis  process,  by  developing  and  deploying 
production  quality  design  and  analysis  software  that  is  adaptable  and 
maintainable,  and  developing  and  deploying  multi-physics  based 
Computational  Engineering  tools  that  exploit  next  generation  high-speed 
computer  resources.  These  tools  have  the  real  potential  to  dramatically  reduce 
the  overall  time  and  effort  required  to  design  and  develop  DoD  systems  and 
products  in  fixed  and  rotary  wing  aircraft,  ships,  propulsion  systems,  antennae, 
and  related  systems,  while  greatly  improving  accuracy  as  well. 
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CREATE  Products 


NPIK 

National  Defense  Industrial  Association 


Air  Vehicles 

DaVinci  -  Rapid  conceptual  design  development  and  initial  system  integration 
Kestrel  -  High-fidelity,  full  vehicle,  multi-physics  analysis  tool  for  fixed-wing  aircraft- 
Release  1 .0  Provides  Rigid  body  CFD  fixed  wing  AV  with  preliminary  aeroelastics 
Helios  -  High-fidelity,  full  vehicle,  multi-physics  analysis  tool  for  rotary-wing  aircraft 
-  Release  1 .0  Provides  highly  accurate  calculation  of  rotorcraft  vortex  shedding 
Firebolt  -  Module  for  propulsion  systems  in  fixed  and  rotary-wing  air  vehicles 
Ships 

RDI  -  Rapid  Design  and  Synthesis  Capability — Partnership  with  ONR  and  NAVSEA 
NESM  -  Ship  Shock  &  Damage-prediction  of  shock  and  damage  effects  -  Release 
0.1  provides  initial  ship  shock  vulnerability  analysis  for  underwater  explosions 
NAVYFOAM  -  Ship  Hydrodynamics-predict  hydrodynamic  performance 
IHDE  -  Environment  to  facilitate  access  to  Naval  design  tools  -  Release  1 .0 
provides  initial  user  interface  for  ship  hydrodynamics 
RF  Antenna 

SENTRI  -  Electromagnetics  antenna  design  integrated  with  platforms  -  Release  1.0 
and  1.5  provides  Initial  RF  antenna  design  and  analysis  with  V&V 

Meshing  and  Geometry 

Capstone  -  Components  for  generating  geometries  and  meshes 
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Other  Computational  Applications 


NPIK 

National  Defense  Industrial  Association 


•Fluid  Dynamics 
•Structural  Mechanics 
•Computational  Chemistry 
•Weather  and  Climate  Prediction 
•Electromagnetics 
•Electronics  Design 
•Signal  Processing 
•Materials  Design 
•Acoustics 

•Space  Weather  Prediction 
•Land  Vehicle  Design 
•Munitions  Design 
•Armor  Design 
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Does  your  organization  now  use  any  physics-based 
modeling  as  part  of  your  design  environment? 


NPIK 

National  Defense  Industrial  Association 


1 .  Not  used  currently 

2.  Some  use  currently 

3.  Significant  use  currently 

4.  Don’t  know 


45% 
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What  is  your  design  domain?  (select  all  that 
apply) 


NPIK 

National  Defense  Industrial  Association 


1.  Ships 

2.  Fixed-wing  aircraft 

3.  Missiles 

4.  Helicopters 

5.  RF  Antennas 

6.  Land  Vehicles 

7.  Avionics  /  Electronics 

8.  Other 
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What  is  the  likelihood  your  organization  will  adopt 
physics-based  modeling  in  the  next  5  years? 


NPIK 

National  Defense  Industrial  Association 


1 .  Certainly  Not  (0%) 

2.  Not  Likely  (1-25%) 

3.  Possible  (26-50%) 

4.  Probable  (51-75%) 

5.  Near  Certain  (76-100%) 


15% 


56% 
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In  what  phases  of  the  product  development  lifecycle  are 
you  currently  using  physics-based  modeling?  (select  all 
that  apply) 


NDIK 

National  Defense  Industrial  Association 


1 .  Concept  development 

2.  Design  /  development 

3.  Manufacturing  /  Production 

4.  Sustainment 

5.  Not  currently  using 


74% 
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In  what  phases  of  the  product  development  lifecycle 
would  physics-based  modeling  be  useful  in  the  future? 
(select  all  that  apply) 


NDIK 

National  Defense  Industrial  Association 


1 .  Concept  development 

2.  Design  /  development 

3.  Manufacturing  /  Production 

4.  Sustainment 

5.  None 


78%  80% 


NDIA  Systems  Engineering  Division 

11/16/2011 


CMMI  /  Physics-Based  Modeling  Conferences  -  201 1 


25 


What  do  you  believe  are  potential  benefits  of  physics- 
based  modeling  in  your  organization?  (Select  all  that 

apply) - 


MPIK 

National  Defense  Industrial  Association 


1 .  Reduced  design  time  and  cost 

2.  Reduced  rework  during 
production 

3.  Improved  product  quality 

4.  Increased  product  innovation 

5.  Improved  product  performance 

6.  Not  sure  of  benefits  at  this  time 


79% 


74% 
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Thank  you  for  your  participation! 


NPIK 

National  Defense  Industrial  Association 

•  To  learn  more  about  CMMI  or  Physics-Based 
Modeling  contact  the  NDIA  SE  Division  chair 

-  Bob  Rassa  (rcrassa@ravtheon.com) 


Please  return  the  interactive  voting  devices! 
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*  Your  job 

*  Experiences  with  process  improvement  and  frameworks 

-(e.g.,  CMM,  CMMI,  ISO9001) 

*  Expectations  for  this  class  (5/5  score?) 
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Introduction 
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The  “Classic” 
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Process-centric 

improvement 

-SEI  CMMI 

-  IS09001 

-  Bellcore 


It  can  work! 

-High  risk  of  failure 
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Goal-problem-centric 

improvement 


Goals  and  problems 
can  be  used  to  scope 
and  sequence  the 
improvement  effort 
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Frameworks 


Frameworks  provide  an 
optional  source  of 
improvement  ideas,  e.g., 

-  Life  cycle 
-SEI  CMMI 

-  IS09001 

-  Bellcore 

In  this  workshop,  either  use: 

-  No  framework 

-Current  organization’s  life 
cycle  and  defined  practices 

-  Published  framework 


(^Processes 


Business 

problems 
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Developing  a  Plan 


— bplanned  process  improvement  is  wishful  thinking.” 

— Watts  Humphrey,  Managing  the  Software  Process 
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Developing  a  Plan 

*  Scope  the  Improvement 

1 .  Establish  plan  ownership 

2.  State  the  major  goals  and  problems 

3.  Group  the  problems  related  to  each  goal 

4.  Ensure  that  the  goals  and  problems  are  crystal  clear  and 
compelling 

5.  Set  goal  priorities 

6.  Derive  metrics  for  the  goals 

*  Develop  an  Action  Plan 

*  Determine  Risks  and  Plan  to  Mitigate 
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1.  Establish  Plan  Ownership 
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*  The  plan  meets  the  owner’s  needs,  e.g., 

-Business  goals  and  problems 

*  The  owner  can  be  a  project  manager,  program 
manager,  senior  manager,  or  division  head 

*  The  primary  owner  *  EPG  or  QA  group 

-  Support  functions  can  share  ownership 


•  Different  individuals  can  be  responsible  for  each 
section  of  the  plan 


EPG  =  engineering  process  group 
QA  =  quality  assurance  group 


©  Copyright  2002-2007  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  2.3 


15 


THE 

PRC 

GRC 

2.  State  the  Major  Goals  and  Problems  -i 

Example  Goals 

1.  Create  predictable  schedules 

2.  Successfully  deliver  product  X 

3.  Reduce  rework 

4.  Improve  the  performance  of  our  core  product 

5.  Keep  customers  happy 

6.  Keep  making  a  profit 
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State  the  Major  Goals  and  Problems  -2 

Example  Problems 

1.  Need  better  requirements.  Requirements  tracking  not  in  place.  Changes  to 
requirements  are  not  tracked;  code  does  not  match  specification  at  test  time. 

2.  Management  direction  unclear  for  product  version  2.3.  Goals  change  often. 

3.  Quality  department  does  not  have  training  in  product  and  test  skills. 

4.  Unclear  status  of  changes. 

5.  Lack  of  resources  and  skills  allocated  to  design. 

9.  Defect  repairs  break  essential  product  features. 

10.  Wrong  files  (for  example,  dynamic  link  libraries)  are  put  on  CD.  Unsure  of  the 
correct  ones. 

11.  Revising  the  project  plan  is  difficult.  Items  drop  off,  new  things  are  added, 
plan  is  out  of  date. 

12.  We  don’t  understand  our  capacity  and  do  not  have  one  list  of  all  the  work  we 
have  to  do. 

13.  Schedule  tracking  and  communication  of  changes  to  affected  groups  is 
poor. 
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3.  Group  the  Problems  Related 

to  Each  Goal  -i 

*  Simplify  the  list  by  grouping  the  problems  that  prevent 
each  goal  from  being  achieved. 


Goal 

Problem 

Problem  Description 

1 .  Create 

predictable 

schedules 

Problem  11 

Revising  the  project  plan  is  difficult.  Items  drop 
off,  new  things  are  added,  plan  is  out  of  date. 

Problem  12 

We  don’t  understand  our  capacity  and  do  not 
have  one  list  of  all  the  work  we  have  to  do. 

Problem  13 

Schedule  tracking  and  communication  of 
changes  to  affected  groups  is  poor. 
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Group  the  Problems  Related 
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Goal 

Problem 

Problem  Description 

2.  Successfully 
deliver  product  X 

Problem  1 

Need  better  requirements.  Requirements 
tracking  not  in  place.  Changes  to  requirements 
are  not  tracked;  code  does  not  match 
specification  at  test  time. 

Problem  2 

Management  direction  unclear  for  product 
version  2.3.  Goals  change  often. 
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Goal 

Problem 

Problem  Description 

3.  Reduce 
rework 

Problem  3 

Quality  department  does  not  have  training  in 
product  and  test  skills. 

Problem  4 

Unclear  status  of  changes. 

Problem  5 

Lack  of  resources  and  skills  allocated  to  design. 

Problem  9 

Defect  repairs  break  essential  product  features. 

Problem  10 

Wrong  files  (for  example,  dynamic  link  libraries) 
are  put  on  CD.  Unsure  of  the  correct  ones. 
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4.  Ensure  That  the  Goals  and  Problems 

Are  Compelling  -i 

If  there  is  a  big  enough  reason,  people  will  change  in  a  heart  beat! 


•  Crisis  •  High  development  cost  •  New  business 

•  Business  loss  ♦  Low  profit  •  Increase  volume 

•  Unhappy  customers  •  Leadership  •  Reduce  costs 
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Ensure  That  the  Goals  and  Problems 

Are  Compelling  -2 

*  Example  goals  that  are  not  compelling: 

-  Document  all  processes. 

-  Develop  a  detailed  life  cycle. 

-  Establish  a  metrics  program. 

*  Example  goals  that  are  more  compelling: 

-  Deliver  product  X  by  Dec  15th. 

-  Increase  product  quality  to  a  maximum  of  10  defects  per  release, 
gaining  back  customers  X,  Y,  and  Z,  and  increasing  our  market  share 
by  1 0  percent. 

-  Reduce  rework  to  5  percent  of  project  effort.  Use  that  time  to  create 
new  product  Y. 

-  Improve  schedule  prediction  to  ±  5-day  accuracy,  eliminating  forced 
cancellation  of  vacations. 
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Making  Existing  Goals  More  Compelling 

*  If  you  are  promoting  an  idea: 

-Ask  WHY  (<idea>)  to  elicit  a  more  compelling  reason 
-e.g.,  WHY  (Level  2)  may  give: 

»  Meet  schedules,  less  rework,  more  sanity,  happier  customers 

*  Make  the  compelling  reason  the  goal,  not  the  process  idea 

-e.g.,  goal:  low  maintenance  OR  formal  inspection? 


If  the  reason  is  not  compelling  enough, 
action  will  probably  not  be  taken! 
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Ensure  That  the  Goals  and  Problems 

Are  Crystal  Clear 


Original  Goals 

Goals  Reworded  for  Clarity 

1.  Create  predictable  schedules 

Meet  all  our  cost  and  schedule 
commitments 

2.  Successfully  deliver  product  X 

Deliver  product  X  by  mm/dd/yy 

3.  Reduce  rework 

Reduce  rework  to  less  than  20  percent  of 
total  project  effort 

4.  Improve  the  performance  of  our 
core  product 

Improve  the  performance  of  our  core 
product  (target  to  be  defined) 

5.  Keep  customers  happy 

Achieve  customer  rating  of  9/10  on  product 
evaluation  form 

6.  Keep  making  a  profit 

Keep  profits  at  15  percent  (and  costs  at  the 
same  level  as  last  year) 

©  Copyright  2002-2007  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  2.3 


24 


5.  Set  Goal  Priorities 


THE 

PRC 

GRC 


Goal 

Relative 
Benefit 
of  Goal 

(1-10) 

Relative 
Cost  of 
Goal 

(1-10) 

Priority 

(Benefit/ 

Cost) 

Phase* 

(1.2,3) 

2.  Deliver  product  x  by  mm/dd/yy. 

10 

4 

2.5 

1 

1.  Meet  all  our  cost  and  schedule 
commitments. 

5.  Achieve  customer  rating  of  9/10  on 
product  evaluation  form. 

3.  Reduce  rework  to  less  than  20  percent 
of  total  project  effort. 

6.  Keep  profits  at  1 5  percent  (and  costs  at 
the  same  as  last  year). 

4.  Improve  the  performa nee  of  ou r  core 
software  product.  (Target  to  be 
defined.) 

*Phase  is  based  on  goal  interdependencies,  logical  ordering,  or  timing 
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*  The  Goal-Question-Metric  (GQM)  approach  from  Basili 
states  that  you: 

-  Define  the  principal  goals  for  your  activity 

-Construct  a  comprehensive  set  of  questions  that,  when 
answered,  helps  assess  where  you  are  relative  to  each 
goal 

-  Define  and  gather  the  data  required  to  answer  these 
questions 
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Goal-Question-Metric  Approach  -i 


Goal 

Questions 

Metrics 

Meet  all  our  cost  and  schedule 
commitments. 

Are  we  spending  the  planned 
number  of  hours  on  the  project 
to  complete  it? 

Are  we  hitting  our  milestones? 

Planned  versus  actual  effort  for 
each  project. 

The  number  of  days  each 
milestone  is  early  or  late. 

Deliver  product  X  by 
mm/dd/yy. 

Are  we  spending  the  planned 
number  of  hours  on  the  project 
to  complete  it? 

Are  we  hitting  our  milestones? 

Planned  versus  actual  effort  for 
each  project  milestone. 

The  number  of  days  each 
milestone  is  early  or  late. 

Reduce  rework  to  less  than  20 
percent  of  total  project  effort. 

How  much  time  do  we  spend 
on  rework  now? 

How  does  this  compare  with 
our  development  time  and  are 
we  improving? 

Percentage  of  project  time 
spent  on  rework. 

How  many  defects  do  we  have 
in  the  product  during  design 
and  coding? 

Defect  density:  Number  of 
defects  found  per  unit  size  of 
work  product  (e.g. ,  number  of 
pages  of  design,  number  of 
lines  of  code). 

am  ri. 
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Goal-Question-Metric  Approach  -2 


Goal 

Questions 

Metrics 

Achieve  customer  rating  of 

9/10  on  product  evaluation 
form. 

How  satisfied  are  they  now? 

Are  we  improving? 

Annual  customer  satisfaction 
survey. 

Keep  profits  at  15  percent  (and 
costs  at  the  same  as  last  year). 

What  is  our  profit? 

Is  it  getting  better  or  worse? 

Annual  net  profit. 
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Using  the  Approach  for  a  Single  Project 

What  is  your  goal? 

Reduce  product  development  cycle  to  six  to  nine  months  for  product  X. 

What  is  preventing  you  from  achieving  the  goal? 

1 .  Changing  requirements. 

2.  Loss  of  resources;  difficult  to  replace  people  with  specialized  skills 
who  leave  the  project. 

3.  Too  many  features  for  the  six-  to  nine-month  development  cycle. 

4.  Poor  quality  of  incoming  code  from  other  groups. 

5.  Inadequate  availability  of  test  equipment. 

6.  Lack  of  visibility  within  each  life  cycle  phase.  It  is  difficult  to  know 
whether  we  are  ahead  or  behind  schedule. 

7.  Don’t  always  have  the  resources  available  to  complete  the  planned 
work. 

8.  Difficult  to  find  defects  early. 
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Using  a  Process  Appraisal  to 
Obtain  a  Problem  List 

*  A  scalable  data  collection  method  for  groups  of  ~5-150 
people,  that  results  in: 

-a  list  of  strengths  and  highest-priority  problems  (&  maturity  rating) 

-  buy-in  for  the  problems 

-  buy-in  for  process  improvement  direction 

*  Surfaces  key  problems  that  might  not  have  been  visible 
before: 

-e.g., communication,  systems  engineering,  PI  implementation 

*  Raises  awareness  of  key  issues  facing  the  organization 

*  Brings  management  and  engineering  together 
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Exercise:  Scope  the  Improvement 


1 .  Form  project  teams 

2.  Determine  the  primary 
business  goals  and  problems 
of  your  group 

-  Simplify  the  list  of  goals  and 
problems  by  grouping  the 
related  problems  under  each 
goal 

-  Verify  that  the  scope  of  your 
improvement  program  is 
compelling 

»  If  not,  ask:  Why  do  I  want  to 
achieve  these  goals? 

3.  Discuss  lessons  learned 


Result: 


What  is  your  goal?  3 

What  is  your  goal?  2 

What  is  your  goal?  ^ 

Reduce  product  development  cycle  to  six  to  nine  months  for  product  X 

What  is  preventing  you  from  achieving  the  goal? 

1.  Changing  requirements 

2.  Loss  of  resources;  difficult  to  replace  people  with  specialized  skills  who  leave 
the  project 

3.  Too  many  features  for  the  six-  to  nine-month  development  cycle 

4.  Poor  quality  of  incoming  code  from  other  groups 

5.  Inadequate  availability  of  test  equipment 

6.  Lack  of  visibility  within  each  life  cycle  phase.  It  is  difficult  to  know  whether  we 
are  ahead  or  behind  schedule 

7.  Don0always  have  the  resources  available  to  complete  the  planned  work 

8.  Difficult  to  find  defects  early 
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Developing  a  Plan 

*  Scope  the  Improvement 

*  Develop  an  Action  Plan 

1 .  Enumerate  actions  using  brainstorming  and  a  process 
framework 

2.  Organize  the  action  plan  based  on  the  goals  and 
problems 

3.  Add  placeholders  for  checking  progress  and  taking 
corrective  action 

*  Determine  Risks  and  Plan  to  Mitigate 
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•  Develop  an  Action  Plan 

1 .  Enumerate  actions  using  brainstorming  and  a  process 
framework 

»  la.  What  actions  are  needed  to  address  the  problems  and 
achieve  the  goals? 

»  1b.  If  a  process  improvement  framework  is  being  used,  which 
elements  will  help  the  problems  and  goals  listed? 

2.  Organize  the  action  plan  based  on  the  goals  and 
problems 

3.  Add  placeholders  for  checking  progress  and  taking 
corrective  action 
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la.  Actions  for  Two  of  the  Problems  -i 


Problem 

What  actions  are  needed  to 
address  the  problems  and 
achieve  the  goals? 

1.  Changing  requirements 

Baseline  the  requirements  before 
design  commences 

Only  allow  changes  to  the 
application  interface,  not  to  the 
kernel  routines 

Improve  the  library  control  system 
to  minimize  version  control  errors 

Investigate  requirements 
management  tools 
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Actions  for  Two  of  the  Problems  -2 


Problem  What  actions  are  needed  to 

address  the  problems  and 
achieve  the  goals? 

3.  Too  many  features  for  the  six-  to  Establish  a  review  process  with 
nine-month  development  cycle  clients  to  negotiate  features  for  a 

six-  to  nine-month  development 
cycle 

Rate  each  feature  based  on  value 
to  the  customer  (1-10  points)  and 
cost  to  develop  (1-10  points) 

Establish  an  incremental  delivery 
plan  to  phase  in  lower  priority 
features 
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1b.  Framework  Elements  for  Two  of  the 

Problems -i  Reworded  for  olarity 


l 


Problem 

Which  elements  will  help  the 
problems  and  goals  listed? 

1.  Changing  requirements 

Develop  an  understanding  with  the 
requirements  providers  on  the 
meaning  of  the  requirements. 

(REQM  spl.1) 

Assign  responsibility  and  authority 
for  performing  the  REQM  process. 
(REQM  gp2.4) 

Track  change  requests  for  the 
configuration  items.  (CM  sp2.1) 

REQM  =  Requirements  Management.  CM  =  Configuration  Management 
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Framework  Elements  for  Two  of  the 

Problems  -2 

Reworded  for  clarity 

i 


Problem 

Which  elements  will  help  the 
problems  and  goals  listed? 

3.  Too  many  features  for  the  six-  to 
nine-month  development  cycle 

Reconcile  the  project  plan  to  reflect 
available  and  estimated  resources. 
(PP  sp3.2) 

Identify  and  analyze  project  risks. 

(PP  sp2.2) 

PP  =  Project  Planning 
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Progress  on  Chosen  Framework  -i 


95% 

map 

to 

Level 

2 


Example  Goals 

1.  Create  predictable  schedules 

2.  Successfully  deliver  product  X 

3.  Reduce  rework 

4.  Improve  the  performance  of  our  core  product 

5.  Keep  customers  happy 

6.  Keep  making  a  profit 

Example  Problems 

1.  Need  better  requirements.  Requirements  tracking  not  in  place.  Changes  to 
requirements  are  not  tracked;  code  does  not  match  specification  at  test  time. 

2.  Management  direction  unclear  for  product  version  2.3.  Goals  change  often. 

3.  Quality  department  does  not  have  training  in  product  and  test  skills. 

4.  Unclear  status  of  changes. 

5.  Lack  of  resources  and  skills  allocated  to  design. 

9.  Defect  repairs  break  essential  product  features. 

10.  Wrong  files  (for  example,  dynamic  link  libraries)  are  put  on  CD.  Unsure  of  the 
correct  ones. 

11.  Revising  the  project  plan  is  difficult.  Items  drop  off,  new  things  are  added, 
plan  is  out  of  date. 

12.  We  dond  understand  our  capacity  and  do  not  have  one  list  of  all  the  work  we 
have  to  do. 

13.  Schedule  tracking  and  communication  of  changes  to  affected  groups  is 
poor. 


Initial 

goals 

and 

problems 
address 
43%  of 
Level  2 


Level  5 


Level  4 


Level  3 


Level  2 


_ 
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Progress  on  Chosen  Framework  -2 
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*  Put  each  to  good  use 

-What  problem  could  it 
solve? 

*  Declare  them  not 
applicable 

-Check  with  your 
appraiser  /  auditor! 

*  Meet  the  letter  of  the  law 
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Action  Plan  Owner: 

Primary  Goal  and 
Intermediate  Goals 

(The  result  you  want) 

Purpose  of  Goal 

(Why  do  you  want  to 
achieve  this  goal?) 

Actions 

Priority 

(*=essential) 

Time 

Estimate 

Who 

PRIMARY  GOAL  1 

PURPOSE  OF 
PRIMARY  GOAL  1 

Small  intermediate 
goal  (based  on  problem 
statement) 

Purpose  of  small 
intermediate  goal 

Action 

1* 

Action 

2* 

Action 

3 

Action 

4 

Next  intermediate  goal 

AAAAAAAAAAAAAAAAAAA/ 

/  V  V  V  V  V  V  V  V  V  V  V  V  V  V  V  V  V  V  V 

Purpose  of  next 
intermediate  goal 

Action 

1* 

Template  is  available  at  www.processgroup.com/bookinfo.htm. 
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Example  Improvement  Plan-i 


Primary  Goal  and 
Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(*=essential) 

Reduce  product 
development  cycle  to 
six  to  nine  months  for 
product  X. 

Deliver  earlier 
than 

competition. 

Manage  changing 
requirements  (based  on 
problem  1). 

Prevent  schedule 
slips  resulting 
from  expensive 
scope  changes. 

Only  allow  changes  to  the  application  interface,  not 
the  kernel  routines. 

1*  j 

Assign  responsibility  and  authority  for  performing 
the  REQM  process. 

2* 

yk 

Check  progress  and  take  corrective  action  . 

- 

Step  3:  Add  placeholder 
for  checking  progress  and 

Improve  the  library  control  system  to  minimize 
version  control  errors. 

Investigate  requirements  management  tools. 

3  1 

taking  corrective  action 

Track  change  requests  for  the  configuration  items. 

4 

Develop  an  understanding  with  the  requirements 
providers  on  the  meaning  of  the  requirements  . 

5 

Baseline  the  requirements  before  design  commences. 

6 
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Primary  Goal  and 
Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(*=essential) 

Set  feature  priorities  for 
a  six-  to  nine  -month 
development  cycle 
(based  on  problem  3). 

Ensure 

commitments  are 
achievable. 

Establish  a  review  process  with  clients  to  neg  otiate 
features  for  a  six  -  to  nine  -month  development  cycle. 

1* 

Rate  each  feature  based  on  value  t  o  the  customer  (1  - 
10  points)  and  cost  to  develop  (1  -10  points). 

2* 

Check  progress  and  take  corrective  action  . 

- 

Reconcile  the  project  plan  to  reflect  available  and 
estimated  resources. 

3 

Identify  and  analyze  project  risks. 

4 

Establish  incremental  delivery  plan  to  phase  in  lower 
priority  features. 

5 
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Choose  Actions  That  Are  Appropriate 

for  the  Problem  -i 


Problem 

Inappropriate  and  Overly 

Complex  Solution 

Unable  to  get  requirements  from 
customers 

'A^opt  quality  function  deployment/’ 

No  time  allocated  for  design 

Adopt  ^detailed  object-oj/fted 
design  probers  / 

Inaccurate  estimates 

Create  a  new  fiistori€al  database, 
built  from  scraJpKTbnd  available  on 
four  platforjjas  \ 

Poor-quality  products 

Define/fetailed  life  cyclebs. 
containing  numerous  engineeili^g 
methods  \ 
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Choose  Actions  That  Are  Appropriate 

for  the  Problem  -2 


Problem 

Simpler  Solution 

Inaccurate  estimates 

Learn  an  estimation  process  that 
addresses  some  of  the  root  causes 
of  the  inaccurate  estimates  (for 
example,  the  Wideband  Delphi 
method) 

Start  collecting  actual  data  for 
current  projects  so  that  they  can 
compare  their  estimates  with  actual 
effort  expended 
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Choose  Actions  That  Are  Appropriate 

for  the  Problem  -3 


Problem 

Simpler  Solution 

Poor-quality  products 

Inspect  (peer  review)  all  critical 
documents  and  code. 

Improve  estimation  of  test  time 
needed. 

Train  test  engineers  in  test  skills. 

Send  test  engineers  to  a  customer 
site  to  understand  how  the 
customer  uses  the  product.  Factor 
this  knowledge  into  the  test 
strategy. 
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Exercise:  Develop  an  Action  Plan 


1 .  Form  project  teams 

2.  Select  2-3  goals  (and  related 
problems)  to  develop  actions 
for 

3.  Develop  actions: 

-  Brainstorming 

-  Select  elements  from  an 
improvement  framework 

-  Establish  priorities  and 
essential  actions 


Result: 


Primary  Goal  and 

Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(^essential) 

Reduce  product 
development  cycle  to 
six  to  nine  months  for 
product  X. 

Deliver  earlier 
than 

competition. 

Manage  changing 
requirements  (based  on 
problem  1). 

Prevent  schedule 
sips  resulting 
from  expensive 
scope  changes. 

Only  allow  changes  to  the  application  interface,  not 
the  kernel  routines . 

1* 

Assign,  responsibility  and  authority  for  performing 
the  REQM  process . 

2* 

Check  progress  and  take  corrective  action  . 

- 

Improve  the  library  control  system  to  minimize 
version  control  errors . 

Investigate  requirements  management  tools. 

3 

Track  change  requests  for  the  configuration  items. 

4 

Develop  an  understanding  with  the  requirements 
providers  on  the  meaning  of  the  requirements 

5 

Baseline  the  requirements  before  design  commences. 

6 

4.  Discuss  lessons  learned 
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Developing  a  Plan 

*  Scope  the  Improvement 

*  Develop  an  Action  Plan 

*  Determine  Risks  and  Plan  to  Mitigate 

1 .  Determine  Scope  of  Risk  Session 

2.  Select  the  Team  and  Moderator 

3.  Identify  Risks 

4.  Analyze  Risks 

5.  Plan  to  Mitigate 

6.  Plan  for  Periodic  Risk  Review 
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A  risk  is  anything  negative  that  could  happen 
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1+2.  Scope,  Team  and  Moderator 

•  Scope: 

-The  complete  list  of  goals  and  problems,  or  subset 

•  Team: 

-  Improvement  team 

-  People  who  have  done  similar  improvement  projects 
and  tasks 

-Technical  experts 

-  Customer  champions 

•  Moderator: 

-Any  team  member 
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3.  Identify  Risks 

Create  a  list  of  potential  future  problems 

•  Weak  areas  such  as  unknown  technology 

-  e.g.,  tools,  vendors,  and  methods  that  are  new  to  the  team 

•  Aspects  that  are  critical  to  the  improvement  project 

-  e.g.,  the  timely  delivery  of  a  vendor’s  training  program, 
continued  buy-in  from  management,  and  the  creation  of 
training  materials 

•  Problems  that  have  plagued  past  projects 

-  e.g.,  loss  of  essential  staff,  resistance  to  change,  and 
changing  priorities 
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4.  Analyze  Risks 

For  each  risk  item: 

-  Does  the  team  understand  this  risk  item? 

»  If  necessary,  split  into  separate  risk  items 

“Lack  of  management  buy-in”  is  replaced  by: 

->  “Jane  could  decide  that  the  new  method  is  not  beneficial” 

->  “Robert  (VP)  does  not  see  any  gains  from  money  spent” 

-  Discuss  and  determine  its  scope: 

»  What  would  the  consequences  be  if  this  risk  item  did  occur? 

-  Determine  what  the  impact  (I)  would  be  if  the  worst 
happened,  using  a  scale  of  one  to  ten. 

-  Determine  how  likely  (L)  it  is  that  the  risk  item  will  occur, 
using  a  scale  of  one  to  ten. 

-  Determine  the  priority  (P)  of  the  risk  items  using  impact  x 
likelihood. 
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Risk  Items 

Consequence 

L 

1 

P  =  Lx  1 

Management  buy-in  for  improvement 
diminishes 

Improvement  program  fails 

9 

10 

90 

Management  changes  priorities  before 
we  complete  any  milestone 

Improvement  program 
loses  credibility 

9 

9 

81 

New  requirements  management  tool 
has  long  learning  curve 

Developers  give  up  in 
frustration 

9 

8 

72 

Library  control  person  might  leave 

Wasted  time  training  a 
new  person 

7 

8 

56 

New  group  to  manage  baseline  changes 
is  not  accepted  by  project  managers 

Duplication  of  effort  or 
baseline  changes  are  not 
managed 

6 

9 

54 

Creation  of  specialized  training 
materials  for  new  staff  takes  too  long 

Improvement 
implementation  delayed 

5 

7 

35 

Requirements  management  tool  is 
delivered  to  us  late 

Pass  up  the  opportunity  to 
try  the  tool 

4 

3 

12 
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5.  Plan  to  Mitigate 

*  Select  the  most  important  risk  issues,  such  as  the  top  2- 
3,  or  top  20% 

*  Brainstorm  on  actions  that  could  be  taken  to  reduce  the 
likelihood  of  the  risk  item  occurring  (risk  mitigation) 

-  Make  actions  specific 

*  Brainstorm  on  actions  that  could  be  taken  to  reduce  the 
impact  if  the  risk  item  does  occur  (risk  contingency 
planning) 

*  Decide  which  actions  to  pursue 

*  Select  a  person  to  be  responsible  for  each  action  chosen 

*  Document  the  information  in  the  risk  management  plan 
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Risk  Planning  Example 

Risk  item: 

•  Management  buy-in  for  improvement  diminishes 

Possible  actions  to  reduce  likelihood  of  risk  item: 

•  Ensure  that  the  improvement  program  addresses  the  management 
team’s  problems  and  goals 

•  Establish  a  steering  committee  to  oversee  the  improvement  effort 

•  Publicize  early  results  to  management 

•  Provide  four  funding  options  for  the  improvement  program:  full¬ 
time,  part-time,  short  bursts,  and  investment  spread  over  two  years 

Possible  actions  to  reduce  impact  of  risk  item  if  it  does  occur: 

•  Determine  improvements  that  can  be  made  at  a  project  level 
without  major  funding 

•  Explain  the  problems  and  goals  that  will  not  be  addressed  because 
of  reduced  funding 
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Risk  Item 

Consequence 
(If  Risk  Item 
Does  Occu  r) 

L 

1 

P 

Actions  to  Reduce 

Likelihood  of  Risk 
Occurring 

Actions  to 
Reduce  Impact  if 
Risk  Does  Occur 

Who  is 
Responsible 
for  These 
Actions? 

When 
Actions 
Should  be 
Complete 

Status 

of 

Action 

Template  is  available  at  www.processgroup.com/bookinfo.htm. 
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Using  the  Risk  Information 


Project  Tasks 
1  Task  A 
2Task  B 
3  Task  C 

5  Task  E 

6  Task  F 


Project  Tasks 


<TTask  D 
2TaskA 
3Task  B 

4  Task  C 

5  Task  E 

6  Task  F 

la.  Task  G 

lb.  Task  H 
5a.  Task  I 


The  improvement  team  should  direct  its  attention  to  the 
higher  risk,  rather  than  the  easier  (lower  risk)  tasks 
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Purpose: 

To  determine  if  the  identified  risks  have  changed  and 
update  the  plan  accordingly 


*  Be  sure  that  periodic  risk  reviews  are  held  to  monitor  the 
risks  identified 

*  Establish  how  often  risks  should  be  reviewed  (once  a 
month  is  typical) 

*  Risk  reviews  can  be  incorporated  into  existing  project 
status  and  phase  reviews 

*  Update  the  list  based  on  risk  review  sessions 
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Exercise:  Determine  Risks  &  Plan  to 

Mit'gate  Result: 


1.  Form  project  teams. 

2.  Determine  the  scope  of  the 
risk  session  and  establish  the 
goal  (i.e.,  which  goals  and 
problems). 

3.  Determine  the  risks  and 
priorities.  Consider  your 
assumptions  and 
improvement  actions. 

4.  Determine  the  action  items  for 
top  2-3  risks. 

5.  Discuss  lessons  learned. 


Risk  Items 

Consequence 

L 

1 

P  =  L  x  1 

Management  buy-in  for  improvement 
diminishes 

Improvement  program 
fails 

9 

1 

0 

90 

Management  changes  priorities 
before  we  complete  any  milestone 

Improvement  program 
loses  credibility 

9 

9 

81 

New  requirements  management  tool 
has  long  learning  curve 

Developers  give  up  in 
frustration 

9 

8 

72 

Library  control  person  might  leave 

Wasted  time  training  a 
new  person 

7 

8 

56 

New  group  to  manage  baseline 
changes  is  not  accepted  by  project 
managers 

Duplication  of  effort  or 
baseline  changes  are 
not  managed 

6 

9 

54 

Creation  of  specialized  training 
materials  for  new  staff  takes  too  long 

Improvement 
implementation  delayed 

5 

7 

35 

Requirements  management  tool  is 
delivered  to  us  late 

Pass  up  the  opportunity 
to  try  the  tool 

4 

3 

12 

Action  Items 
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Summary  -  Developing  a  Plan 

*  All  improvements  are  tied  to  specific  needs  of  the 
organization 

*  Goals  and  problems  help  the  organization  identify 
which  pieces  of  an  improvement  framework  to 
implement  next 

*  Goals  and  problems  establish  the  scope  and  context 
for  each  improvement 

-When  a  problem  has  been  solved  or  a  goal  addressed, 
a  team  can  stop  defining  the  process  or  standard 

*  Practitioners  and  managers  are  motivated  to  work  on 
improvement  because  the  effort  is  directed  toward  the 
group’s  needs 
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Implementing  the  Plan 


— Pnring  that  the  true  skeptics  are  indeed  truly  skeptical  achieves  nothing, 
except  that  you’ve  dented  your  pick  and  probably  permanently  diminished 
your  credibility  (and  failed  to  appreciate  the  vital  importance  of  building  a 
fragile  momentum).” 

— Tom  Peters,  A  Passion  for  Excellence 
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*  A  (big)  process  document  is  written 

*  The  improvement  team  assumes  it  is 
done  and  deployment  is  “ji 
to  the  people” 

*  The  process  is  “deployed” 

*  The  process  is  ignored,  or  significant 
resistance  occurs 

*  The  organization  gives  up  or 
continues  to  struggle 
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-Sell  Solutions  Based  on  Needs 
-Work  with  the  Willing  and  Needy  First 
-  Keep  Focused  on  the  Goals  and  Problems 
-Align  the  Behaviors  of  Managers  and  Practitioners 
-Avoiding  a  Documentation  Glut 
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The  Selling  Aspect  of  Getting 
People  to  Change 


*  What  did  the  sales  person  do  in  your  best  sales 
experience? 
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Individuals  Want  to  be  Understood  First 
and  Then  Have  Their  Problems  Solved 


“And  I  say  you  can  afford  it!" 
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PROCESS 


How  to  Use  Selling 

•  Forget  what  you  are  selling 

•  Understand  what  the  customer 
wants  in  his/her  terms 

-  Problems  and  goals 

•  Determine  the  match  with  what  you 
have  and  what  the  customer  wants 

•  Solve  the  customer’s  problem 

-  may  be  a  standard  or  customized 
solution 
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-Sell  Solutions  Based  on  Needs 
-Work  with  the  Willing  and  Needy  First 
-  Keep  Focused  on  the  Goals  and  Problems 
-Align  the  Behaviors  of  Managers  and  Practitioners 
-Avoiding  a  Documentation  Glut 
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Work  with  the  Willing  and  Needy  First 


*  A  planned  and 
staged  approach: 

-Builds  momentum 

-  Leverages 
success  stories 

-  Provides 
feedback  to  refine 
the  solution(s) 

-  Easier  to  manage 
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What  Stages? 


3.  Early  Majority 
People  that 
need  evidence 


4.  Late  Majority 
Heavy  skeptics 


2.Earlv  Adopters 
People  that 
are  almost 
ready 


1.  Innovators 


Change  for 
change 
sake 


Need  & 
Timing 


•  No  perceived 
problem  to  solve 

•  Neither  angry  or 
seducible 

•  Doesn’t  think 
management  is 
serious 


Waiting 


5.Laqqards 


Mistrust 


Kill  me 


Time 
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▲ 


1 .  Interview  to  gather 
needs 

-  By  department, 
project  team  or 
individual 


2.  Sort  interviewees  by 

-  Need  for  the 
solution 

-Willingness  to  try 
the  solution 


Change 
now 


Don’t  know.they  need  it 


Need  & 
Timing 


No  need 
unwilling 
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Three  Uses  of  the  Adoption  Curve 


1 .  Increase  the  speed  of  deployment  by  determining  with 
whom  to  work  and  in  which  order 

2.  Reduce  the  risk  of  failure  by  building  and  deploying 
the  solution  in  increments 

3.  Determine  when  to  develop  a  policy  and  issue  an 
edict 
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Use  1 :  Increase  Speed  of  Deployment 

Speed  comes  from: 

-  Increasing  motivation  to  adopt  -  based  on  need 

-  Decreasing  resistance  -  based  on  willingness 

-  Using  previous  successes  to  influence  the  next  group 


Successes:  Getting  the  Word  Out 
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*  How  to  reach  your  audience 


Written  promotional 

material 

brochures,  articles, 
reports,  newsletters. 


Spoken  word 

brown  bags,  seminars, 
department  meetings, 
discussion  groups 


Written  processes 

job  aids,  guidelines 
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Use  2:  Reduce  the  Risk  of  Failure 


Innovators  and  early  adopters  can 
provide  specific  requirements  for 
and  feedback  on  early  versions  of 
the  solution 


Change 


For  example: 

Planning  solution  for  3  teams 

•  Team  A:  Estimation 

•  Team  A:  Negotiation 

•  Team  B:  Risk  management 

•  Team  C:  Metrics  /  tracking 
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Use  3:  Developing  a  Policy 
&  Issuing  an  Edict  -i 
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NOW 


Policy  and  edict  time  =  -50% 


Time 


Version  2.3 
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•  Policy  states: 

-When  and  where  a 
practice  should  be  used. 

*  In  the  beginning: 

-You  might  not  have  any 
idea! 


Wait  until  you  get  some 
experience  and  feedback 


*  Edict  states: 

- Ddt  now,  this  is 

important.” 

*  In  the  beginning: 

-You  don’t  necessarily 
have  proof  or  credibility. 


Wait  until  you  get  some 
experience  and  ownership 
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3. Ear 


People  that 
need  evidence 


2. Early  Adopte 


People  that 
are  almost 
ready 

1  .Innovators 
Change  for 
change 
sake 


Need  & 
Change  Timing 


v  Majority 


4 .Late  Majority 
Heavy  skeptics 


No  perceived 
problem  to  solve 

•  Neither  angry  or 
seducible 

*  Doesn’t  think 
management  is 
serious 


Waiting 


5.Laqqards 


Mistrust 


Kill  me 


Time 
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Negative  Reaction  to  Change 


Energy 


AngerLRage 

w 
k 


Bargaining 


Denial 


Stunned  Paralysis 


Acceptance 


Depression 


-*  Time 

78 
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Resistance  has  two  common  causes: 

1 .  It  is  not  apparent  to  the  person  resisting  that  your 
solution  will  meet  his  or  her  current  needs 

2.  The  person  believes  that  your  solution  brings  more 
pain  than  benefit;  examples  of  pain  include: 

-  embarrassment  (if  the  change  is  unsuccessful) 

-  wasted  time  using  a  poorly  constructed  solution 

-  fear  of  stepping  into  the  unknown  (when  the  status  quo 
is  comfortable) 
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Overcoming  Resistance  -  Needs -i 

*  Address  the  first  common  cause  by: 

-  identifying  and  clarifying  the  needs  of  your  audience 

»  What  is  the  problem  and  what  are  they  trying  to  accomplish? 

»  Do  they  understand  your  proposed  solution  and  is  this  an  appropriate 
time  to  adopt  the  idea? 

»  What  are  their  concerns  regarding  costs  (or  effort  /  timing)? 

*  If  your  solution  does  not  match  the  need,  then  say  so, 
and  investigate  other  solutions  that  do 

*  If  the  issue  is  timing  or  cost: 

-determine  a  more  appropriate  occasion  to  deploy  the 
new  skill,  or 

-  propose  a  smaller,  more  economical  solution 
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Overcoming  Resistance  -  Needs  -2 

Understand  the  benefit  of  the  current  behavior.  The  new 
behavior  must  give  them  at  least  the  benefit  of  the  old. 


Example: 

*  Quarterly  product  releases: 

-  Keeps  customers  happy 

-  Helps  keep  team  focused 

-  Generates  constant  revenue 

*  Semi-annual  releases  may  also: 

-  Keeps  customers  happy 

-  Helps  keep  team  focused 

-  Generates  constant  revenue 
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Adoption  Requires  Leverage 

(Some  identified  pain,  or  missing  pleasure,  associated  with 

the  current  behavior) 


“I’ll  do  this  peer  review 
stuff  if  it  can  help  me 
make  tomorrow’s 
demonstration  for  the 
CEO” 


Unless  you  find  that  area  of  leverage  (pain  or 
pleasure),  the  change  may  never  happen 


©  Copyright  2002-2007  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  2.3 


82 


THE 

PRC 

GRC 

Overcoming  Resistance  -  Beliefs  -i 

Events  lead  to  beliefs  (things  we  feel  certain  about).  Beliefs 
combined  with  values  (what  is  important)  lead  to  behaviors. 


Event  or  Information 

Bad  code  review 


Beliefs 

Reviews 
grade  the 
author 


Values 

•Deadline 

•Ego 

•Respect 
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Overcoming  Resistance  -  Beliefs  -2 

*  Understand  customer’s  values 

•  What  is  most  important  to  you? 

•  What  is  most  important  to  you  about  planning? 

*  Understand  beliefs 

•  What  have  you  heard  about  code  reviews? 

•  What  have  your  experiences  been  with  process 
improvement? 

*  Use  discussion,  new  information  and  events  to  help 
correct  any  inaccurate  beliefs 

•  For  example: 

-ensure  that  the  trial  of  a  new  idea  is  successful 
-use  testimonials 
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Plan  to  Reduce  Resistance 

Internalization 
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Accelerating  Adoption  Through 
Training  and  Consulting  -i 


2. Ear 


are 


I— 1 

P 

n 

/ 

£ 

l( 

c 

T 

Ir 

j 

5t 

3.  Early  Majority 

Pa^rilQ  thot 


1  .Innovators 
Change  for 
change 
sake 


Need  & 
Change|  Timing 


No  perceived 
problem  to  solve 
Neither  angry  or 
seducible 
Doesn’t  think 
management  is 
serious 


5  .Laggards 


Waiting 


Mistrust 


Time 
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Accelerating  Adoption  Through 
Training  and  Consulting  -2 


Coach 


Facilitator 


/  teacher 


•  Change  Agents:  get  out  of  your 
office  and  help! 

•  Always  work  on  real  problems 

•  Get  other  process  champions 
to  help  you  deploy  solutions 


Politician 
/  go-between 
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Process  Champions  and 
Train-the-trainer 

*  Process  Champion  characteristics: 

-  Highly  respected  (by  the  target  audience) 

-Some  dedicated  time  to  help 

-Able  to  share  information  practically 

*  Train-the-trainer  guidelines: 

-  Provide  the  champion  with  materials 

-  Observe  the  champion  train  an  audience 

-Keep  training  to  small  (2-4  hr),  manageable  topics,  e.g., 

»  Risk  management 
»  Inspection 
»  Estimation 
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-Sell  Solutions  Based  on  Needs 
-Work  with  the  Willing  and  Needy  First 
-  Keep  Focused  on  the  Goals  and  Problems 
-Align  the  Behaviors  of  Managers  and  Practitioners 
-Avoiding  a  Documentation  Glut 
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Keep  Focused  on  the  Goals  and 

Problems  -i 


Standards,  standards,  standards 


Tools,  tools,  tools 


Code,  code,  code 


(e.g.,  project 
goals  achieved, 
project  problems 
solved,  improved 
skills  internalized 
for  the  next 
project) 


Ideal  organizational  change 
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Keep  Focused  on  the  Goals  and 

Problems  -2 
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*  Review  improvement  plans  periodically  during 
departmental  and  project  meetings. 

-If  you  are  not  making  weekly  gains  in  your  improvement 
program,  you  may  be  offtrack.  Weekly  gains  come  from 
fixing  numerous,  small  project-level  problems. 

*  Are  improvement  activities  tied  to  the  business  goals 
and  problems  experienced  by  the  organization? 

*  Are  projects  getting  better  results? 

-e.g.,  improved  customer  satisfaction,  less  rework,  fewer 
surprises,  fewer  communication  problems,  meeting 
deadlines  etc. 
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Keep  Focused  on  the 
Goals  and  Problems  -3 


Don’t  revert  back  to 

process-centric 

improvement 

-  One  framework 
topic  at  a  time 

Practice  and 
internalize  goal- 
problem  approach 


Business 

goals 


Business 

problems 
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Keep  Focused:  Doing  Too  Much  at  Once 


•  Monday’s  plan 


Primary  Goal  and 

Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(*=essenti  al) 

Reduce  product 
development  cycle  to 
six  to  nine  months  for 
product  X 

Deliver  earlier 
than 

competition 

Manage  changing 
reqiirements  (based  on 
problem  1). 

Prevent  schediie 
slips  resulting 
from  expensive 
scope  changes. 

Only  alio  w  changes  to  the  appli  cation  int  erface,  not 
the  Kernel  roilin  es. 

1* 

Establish  a  group  with  the  authority  for  managing  the 
project's  software  baselin  es. 

2* 

Check  process  and  take  corrective  action. 

Improve  the  library  control  system  to  minimi  ze 
version  control  errors. 

Investigate  reqiirements  management  tools  . 

3 

Record  and  track  change  requests  and  problem 
reports  for  all  configuration  items. 

4 

Review  the  initi  al  requirements  and  changes  before 
they  are  in  corporate  cl  into  the  project  plan. 

5 

Baselin  e  the  requirements  before  design  commences. 

6 

*  Tuesday’s  plan 


Primary  Goal  and 

Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(*=essenti  al)  ' 

Set  feature  prioriti  es  for 
a  six-  to  nin  e- month 
development  cycle 
(based  on  problem  3). 

Ensure 

commitm  ents  are 
achievable. 

Establish  a  review  process  with  clients  to  negotiate 
features  for  a  six-  to  nin  e-month  development  cycle 

1* 

Rate  each  feature  based  on  value  to  the  customer  (1- 
10  points)  and  cost  to  develop  (1-10  points ). 

2* 

Check  progress  and  take  corrective  action . 

- 

Review  project  commitm  ents  with  senior  managers, 
engineers  and  the  customer  to  obtain  agreement 

3 

Perform  risk  management  related  to  the  schedule, 
resource  and  techni  cal  aspects  of  the  project. 

4  i 

5 

Establish  in  cremental  deliv  ery  plan  to  phase  in  lo  wer 
priority  features. 

5 
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Keep  Focused:  Doing  Too  Much  at  Once 

-2 

*  Success  in  any  discipline  is  accomplished  by  focusing 
on  a  few  items  at  a  time 

-  Stick  to  the  priorities  you  established  in  your  plan 

*  See  a  few  improvements  to  completion  so  the 
organization  can  experience  success 

*  Early  successes  provide  fuel  and  motivation  to 
address  the  remaining  goals  and  problems 
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-Sell  Solutions  Based  on  Needs 
-Work  with  the  Willing  and  Needy  First 
-  Keep  Focused  on  the  Goals  and  Problems 
-Align  the  Behaviors  of  Managers  and  Practitioners 

-Avoiding  a  Documentation  Glut 
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What  Management  Can  Do  -i 

•  At  all  management  levels: 

-  Place  improvement  in  top  five  priorities 

-  Communicate  PI  for  progress  on  business  goals  and 
problems 

»  PI  is  not  documentation  for  some  appraiser!  (Documentation  captures  a 
solution  to  a  problem) 

-  Maintain  overall  department  improvement  plan  and 
track  publicly 

-Ensure  that  improvement  actions  are  in  current  plans 

»  Fix  current  project  problems  as  we  go 
»  Mitigate  current  project  risks  as  we  go 

»  Include  time  in  schedules  for  improvement  +  use  allocated  rework  time 

-Lead  by  example  (PP,  CM,  REQM,  vendor  selection) 
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What  Management  Can  Do  -2 

*  At  all  management  levels: 

-  Maintain  consistency  of  purpose 

»  Avoid  flavor-of-the-month 
»  Internalize  the  use  of  process  models 
»  Understand  how  this  practice  helps  us 

-Establish,  track  and  use  measures 

»  E.g.,  customer  satisfaction,  defect  density,  planned  vs.  actual  time 

-  Provide  forums  for  sharing  good  practices 
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What  Management  Can  Do  -3 

Reinforcement  and  reward  will  condition  a  new  behavior 
long-term.  The  reward  must  be  timely  and  meaningful. 

*  Meaningful:  valuable  to  the  person  or  group  getting  the 
reward. 

*  Timely:  as  soon  as  possible  after  the  event. 

*  Don’t  reward  firefighting  or  heroism,  unless  that  is  a 
long-term  goal! 


Examples: 

•  Benefit  from  the  improvement  •  Money 

•  Recognition  •  Free  time 

•  Increased  responsibility 
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What  Management  Shouldn’t  Do 

*  At  all  management  levels,  DON’T: 

-Manage  PI  as  an  activity  completely  unrelated  to 
running  the  business  -  a  documentation  exercise! 

-  Make  the  :PG  solely  responsible  for  achieving  CMMI 
Level  N 

-Undermine  improvements 

»  e.g.,  — ForgteCM,  E-mail  the  product  now!” 

-  Overcomplicate  solutions 

»  25  metrics 

»  Globally  available  historical  database,  15  countries 
»  Every  CMMI  practice  uniquely  met,  instead  of  merging  them  together 
»  Documentation  =  20  pages  per  process 
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-Sell  Solutions  Based  on  Needs 
-Work  with  the  Willing  and  Needy  First 
-  Keep  Focused  on  the  Goals  and  Problems 
-Align  the  Behaviors  of  Managers  and  Practitioners 
-Avoiding  a  Documentation  Glut 
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1.  Focusing  on  the  organization’s  needs 

2.  Keeping  processes  concise 

3.  Knowing  when  you  are  in  trouble 

4.  Knowing  if  you  are  meeting  the  intent  of  the 
framework  (e.g.,  CMMI  process  areas) 
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1.  Focus  on  the  Organization’s  Needs 


QuickTime™  and  a 
TIFF  (LZW)  decompressor 
are  needed  to  see  this  picture. 


Keep  process 
documentation  concise  by 
focusing  it  on  specific  needs 
(e.g.,  business  goals  and 
problems) 

Begin  with  a  simple  version 
of  the  process.  When  the 
need  is  addressed,  stop 

-  Refine  further  when  the 
process  no  longer  meets 
the  need 
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Focus  on  the  Organization’s  Needs 

Example 


Project  Needs 

SEI  CMMI  Practices  That  Would  Help 

Changing  requirements. 

^  Level  2:  PP  -  Specific  Practice  2.1 

r  Establish  and  maintain  the  projectO&udget 
'  and  schedule. 

The  poor  quality  of  incoming  code  from 
other  groups. 

We  routinely  over  commit.  \ 

Inadequate  availability  of  test 
equipment. 

Level  2:  PP  -  Specific  Practice  3^2 

Reconcile  the  project  plan  to  reflect 
available  and  estimated  resources. 

Too  many  features  are  required  for  the 

6-  to  9-month  development  cycle. 

Difficult  to  find  defects  early. 

Use  the  need(s)  to  scope  the  process 
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27  person  wks 
(12  weeks) 


66.5  calendar  weeks 


1 .  Determine  task  dependencies.  2.  Add  task  EFFORT  estimates. 

3.  Add  resources  -  people,  equipment,  resource  assumptions. 

4.  Add  resource  availability  -  %allocation,  calendar  days  out. 
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Example  Process  for  Reconciling  Commitments 

(Level  2:  PP  -  Specific  Practice  3.2) 


Step  1 :  Project  team  determines  high-level  product  needs  (or 

scope  of  work),  from  customer  and  marketing  input 

Step  2:  Project  team  develops  an  initial  project  plan  and 
estimates  to  determine  what  is  feasible 

Step  3:  Project  team  meets  with  management,  marketing, 

customers  and  related  groups  to  determine  whether: 

-  the  change  or  product  is  feasible 

-  there  is  agreement  to  the  resource,  cost  and  schedule  estimates 

-  the  risk  is  acceptable 

Step  4:  A  commitment  is  made  OR  further  negotiation  is  held 
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2.  Keep  Processes  Concise  -i 


Always  consider  1  page  (small)  for 
each  process  or  sub  process! 

-  Refine  what  you  have  defined,  don’t 
necessarily  add  more 

A  Defined/Managed  Process  can  be 
the  instructions  embedded  in  a  work 
product  template;  e.g., 

-  The  template  for  an  CM,  QA  or  project 
plan 

A  standing  agenda  can  be  the 
process  for  a  project  review 

-  With  instructions  for  use 


CM  Plan  Template 

1 .  List  Configuration  Items 

x,  y  z 

2.  Establish  File  Naming 

Convention 

File-x<n>.doc 

3.  Establish  Baseline  File 

Structure 

***  ***  *** 

/| 
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Example  Milestone  Review  Process 

(Level  2:  Project  Monitoring  &  Control  -  Specific  Practice  1 .7) 


For  the  last  period: 

-  The  original  plan 

-  Accomplishments 

-  The  critical  path  of  the  project 

-  High-risk  areas  that  need  attention  (top  2-3) 


-  Problems  that  are  impacting  quality,  cost  and  the  schedule 

-  Status  of  action  items  (open  and  closed) 


*  For  the  next  period: 
-  The  plan 


Instructions  for  use: 
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Keep  Processes  Concise  -3 

•  Don’t  have  separate  QA  checklists  that  repeat  the 
original  process.  Use  the  original  process  as  the 
checklist. 

-Add  specific  QA  pointers  and  guidelines. 

*  Look  for  reuse  in  your  CMMI  implementation,  e.g., 


CMMI  Practice 

This  CMMI  Practice  Can 
be  Used  Here  Too 

Level  2:  PP  -  Specific  Practice  1.1 

Establish  a  top-level  work  breakdown 
structure  (WBS)  to  estimate  the  scope 
of  the  project 

GP  2.2  in  all  Process  Areas 

Plan  the  Process 
+  use  PA  process  description 

Level  3:  VER  -  Specific  Practice  2.2 

Conduct  Peer  Reviews 

Level  3:  TS  -  Generic  Practice  2.9 

Evaluate  adherence  of  the  technical 
solution  process 

VER  =  Verification,  TS  =  Technical  Solution 
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Keep  Processes  Concise  -4 


Consider  one  representation 

-  e.g.,  PowerPoint  can  be 
printed,  shared  and 
presented 

Embed  tailoring  guidelines 

-  A  separate  document  can 
be  difficult  to  find  and 
update 

Have  one  policy 

-  e.g.,  — Perfon  the  life  cycle 
Follow  the  tailoring 
guidelines  in  the  lifecycle 


Estimation  Process 

Step  5:  Use  the  historical  database  to  verify 
the  estimate  for  each  task 

Purpose:  To  search  the  organization's  historical  data 
to  see  if  a  similar  task  (or  group  of  tasks)  exists. 

Tailoring  guideline :  This  step  should  be  performed 
whenever  applicable  data  exists.  It  can  be  discarded 
when  a  new  language  or  technology  is  being  used. 

Risk  if  omitted:  Failure  to  use  the  database  could 
result  in  significant  oversight  about  schedule 
estimates,  and  could  lead  to  a  loss  in  revenue. 

Minimum  requirement:  Data  that  exists,  but  is  not 
considered  applicable  for  the  current  estimate,  must  be 
reviewed  with  one  other  manager  to  verify  non- 
applicability. 
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3.  Know  When  You  are  in  Trouble 


•  Project  team  members  create 
process  and  project 
documentation  to  please  an 
appraiser  or  auditor 


QuickTime™  and  a 
Photo  -  JPEG  decompressor 
are  needed  to  see  this  picture. 


•  It  has  been  6  months  and  still  the 
process  is  not  ready  to  use 


•  Project  managers  -study”  the 
documentation  in  preparation  for 
the  appraisal 


•  The  ink  refuses  to  dry,  and  the  appraisal  interview  is  about  to  start! 
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4.  Know  if  You  are  Meeting  the  Intent  of 

the  framework 

(e.g.,  CMMI  Process  Areas) 

*  The  problems  related  to  those  Process  Areas  have 
been  solved  and  the  solutions  are  captured  in  the 
processes. 

*  Project  and  process  documents  are  used  to  run  the 
project  and  the  business. 

-The  practices  within  the  CMMI  have  been 
institutionalized.  The  process  — livaS 

-  No  — etxa  paperwork”. 

*  The  processes  have  “institutionalized”  characteristics. 

-  E.g.,  documented,  planned,  resourced,  trained,  someone 
assigned,  under  control,  meet  needs,  monitored. 
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Process  documentation  is: 

*  Only  a  small  part  of  process  improvement 

*  A  method  of  capturing  and  sharing  engineering 
and  management  practices 
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Summary:  Implementing  the  Plan 

*  Don’t  go  after  the  hardest  nut  (laggard)  first 

*  Focus  on  real  needs  (who  needs  what,  when) 

*  The  process  provider  needs  to  be  flexible  and  provide 
appropriate,  timely  solutions 

*  PI  is  not  about  documentation 

*  Management  can  lead 
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Exercise:  Implementing  the  Plan 


1.  Make  notes  on  which  project  members  are  the  innovators 
and  early  adopters,  and  what  they  are  ready  to  adopt.  Refine 
your  current  improvement  action  plan  using  this 
information.  What  other  interviews  do  you  need  to  conduct? 

•  Also,  consider  the  late  majority  (skeptics)  and  laggards. 

2.  Review  your  improvement  plan:  Is  it  aimed  at  deploying 
solutions  in  small  pieces  based  on  project  needs  and 
priorities? 

•  For  example,  a  complete  project  planning  process  can  be 
broken  into  estimation,  negotiation,  risk  identification,  and 
scheduling. 

3.  Develop  a  plan  to  interview  some  managers  and  find  one 
who  can  use  some  of  the  techniques  you  are  advocating, 
such  as: 

•  Estimating,  planning  or  peer  reviews. 

4.  Identify  someone  who  can  help  deploy  well-tried 
improvements  in  parallel  with  your  efforts. 


Result: 


Action  Items 
1. - 

2 


3. 


4. 
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Checking  Progress 


— Yoi&an  design  a  measurement  system  for  any 
conclusion  you  wish  to  draw.” 

— Gerald  Weinberg,  Quality  Software  Management 
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•  Checking  progress: 

-  lets  you  know  how  well  your  improvement  program  is  going 

-  provides  visibility  to  detect  problems  early 

-gives  you  data  to  make  your  future  plans  more  effective 


*  Corrective  action  consists  of: 

-  mid-course  changes  based  on  results  and  lessons  learned 
from  the  planning  and  implementation  phases 
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Goal-Question-Metric  Approach  -i 


Goal 

Questions 

Metrics 

Meet  all  our  cost  and  schedule 
commitments. 

Are  we  spending  the  planned 
number  of  hours  on  the  project 
to  complete  it? 

Are  we  hitting  our  milestones? 

Planned  versus  actual  effort  for 
each  project. 

The  number  of  days  each 
milestone  is  early  or  late. 

Deliver  product  X  by 
mm/dd/yy. 

Are  we  spending  the  planned 
number  of  hours  on  the  project 
to  complete  it? 

Are  we  hitting  our  milestones? 

Planned  versus  actual  effort  for 
each  project  milestone. 

The  number  of  days  each 
milestone  is  early  or  late. 

Reduce  rework  to  less  than  20 
percent  of  total  project  effort. 

How  much  time  do  we  spend 
on  rework  now? 

How  does  this  compare  with 
our  development  time  and  are 
we  improving? 

Percentage  of  project  time 
spent  on  rework. 

How  many  defects  do  we  have 
in  the  product  during  design 
and  coding? 

Defect  density:  Number  of 
defects  found  per  unit  size  of 
work  product  (e.g. ,  number  of 
pages  of  design,  number  of 
lines  of  code). 

am  ri. 

jhWfefctvig.  OUr  CUrrefftkprocessgroup.coni 
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Goal-Question-Metric  Approach  -2 


Goal 

Questions 

Metrics 

Achieve  customer  rating  of 

9/10  on  product  evaluation 
form. 

How  satisfied  are  they  now? 

Are  we  improving? 

Annual  customer  satisfaction 
survey. 

Keep  profits  at  15  percent  (and 
costs  at  the  same  as  last  year). 

What  is  our  profit? 

Is  it  getting  better  or  worse? 

Annual  net  profit. 
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-  Are  We  Making  Progress  on  the  Goals? 

-  Are  We  Making  Progress  on  Our  Improvement  Plan? 

-  Are  We  Making  Progress  on  the  Improvement  Framework? 

-  What  Lessons  Have  We  Learned  So  Far? 
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Observations  and  Corrective  Actions 

*  Tracking  the  data  for  each  project  showed: 

-a  trend  of  consistently  underestimating  the  number  of 
hours  needed 

-that  although  the  group  met  the  majority  of  their  deadlines, 
the  hours  expended  to  do  so  were  causing  some  financial 
loss 

*  Corrective  actions: 

-develop  a  spreadsheet  of  historical  information  and  use  it 
when  estimating  new  projects 

-  use  the  Wide  Band  Delphi  technique  for  deriving  estimates 

-  share  the  data  with  the  sales  staff  to  develop  a  joint 
understanding  of  how  to  bid  on  future  projects 
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Goal:  Reduce  Rework  to  Less  Than 
20  Percent  of  Total  Project  Effort -i 
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Observations  and  Corrective  Actions 

*  The  graph  showed: 

-a  trend  of  improvement  in  how  engineering  time  was  used 

-that  further  improvements  were  necessary  to  achieve  the 
goal 

*  Corrective  actions: 

-  A^B:  effort  estimation,  risk  management,  schedule 
creation,  project  tracking,  inspection  of  design  documents 

-B^C:  inspecting  code  and  requirements  documents, 
formal  CM,  improved  testing,  process  assurance,  post¬ 
project  sessions  on  lessons  learned 

-D^iplan  to  adopt  use  cases  and  design  process 
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Goal:  Reduce  Rework  to  Less  Than  20 


Da 


Java/C++ 


nt  r\f  T/\^l _ PKAinni'  Cff/\i4 _ « 


Java/C++  Inspections  -  Severity  1  +  Severity  2  Defects 
per  Thousands  of  Lines  of  Code  (KLOC) 

Inspections  -  Severity  1  +  Severity  2  Defects  per  Thousands  of  Lines  of  Cod 


Module  1  Module  2  Module  3  Module  4  Module  5  Module  6  Module  7 

(after  unit  (after  release)  (after  release)  (after  release)  (after  unit  (after  release)  (after  unit 
test)  test)  test) 

Inspection  Session 

Inspection  Session 
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Observations  and  Corrective  Actions 

•  Defect  density  of  released  and  tested  software  is 
extremely  high 

-a  cause  of  chaos  and  70%  rework 

*  Corrective  actions: 

-  inspect  a  larger  portion  of  current  code  base 

-develop  common  errors  checklist  to  capture  coding 
mistakes 
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Goal:  Reduce  Rework  to  Less  Than  20 
Percent  of  Total  Project  Effort  -3 


Code  Inspection  Defect  Dei 

6.0t - 


1  3  5  7  9  11  13  15  17  19  21  23  25  27  29  31  33  35  37 


Module 
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Observations  and  Corrective  Actions 

•  Defect  density  of  released  and  tested  software  is 
extremely  low 

-only  1-3%  of  project  effort  spent  on  rework 

*  Corrective  actions: 

-continue  to  inspect  all  work  products 
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-Are  We  Making  Progress  on  the  Goals? 

-Are  We  Making  Progress  on  Our  Improvement  Plan? 

-Are  We  Making  Progress  on  the  Improvement 
Framework? 

-What  Lessons  Have  We  Learned  So  Far? 
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Are  we  Making  Progress  on  Our 

Improvement  Plan? 

Total 

number  of 
goals  and 
intermediate 
goals 
completed 
in  action  12 
plan 

10 
8 
6 
4 
2 


2  4  6  8  10  12  14  16  18  20  22  24 

Month 


Trend  diagram  tracking  goal  and  intermediate  goal  completion 
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Observations  and  Corrective  Actions 

*  No  way  of  meeting  initial  deadline  of  16  months 

*  Corrective  actions: 

-  revise  completion  date  to  24  months 
-dedicate  more  time  for  improvements 

-  use  successes  from  early  adopters  to  motivate  others 

-ensure  that  the  solutions  we  are  developing  do  not  exist 
somewhere  else 
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-Are  We  Making  Progress  on  the  Goals? 

-Are  We  Making  Progress  on  Our  Improvement  Plan? 

-Are  We  Making  Progress  on  the  Improvement 

Framework? 

-What  Lessons  Have  We  Learned  So  Far? 
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Are  We  Making  Progress  on  the 
Improvement  Framework?  -i 

Method  1 :  Count  actions  that  are  from  the  framework 


Primary  Goal  and 

Intermediate  Goals 

(The  results  you  want) 

Purpose  of  Goal 

(Why  do  you 
want  to  achieve 
the  goal?) 

Actions 

Priority 

(*=essential) 

Reduce  product 
development  cycle  to 
six  to  nine  months  for 
product  X 

Deliver  earlier 
than 

competition. 

Manage  changing 
requirements  (based  on 
problem  1). 

Prevent  schedule 
slips  resulting 
from  expensive 
scope  changes. 

Only  allow  changes  to  the  application  interface,  not 
the  kernel  routines. 

i* 

/ 

Assign  responsibility  and  authority  for  performing 
the  REQM  process. 

2*  v 

Check  progress  and  take  corrective  action  . 

- 

Improve  the  library  control  system  to  minimize 
version  control  errors. 

Investigate  requirements  management  tools. 

3 

J 

Track  change  requests  for  the  configuration  items. 

4  V/ 

Develop  an  understanding  with  the  requirements 
providers  on  the  meaning  of  the  requirements  . 

5  V 

Baseline  the  requirements  before  design  commences. 

6 
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Are  We  Making  Progress  on  the 
Improvement  Framework?  -2 

Method  2:  Conduct  a  mini-assessment  to  establish  adoption  of  practices* 

Purpose: 

*  To  evaluate  improvement  progress 

and  make  necessary  adjustments 

Method: 

*  Develop  a  checklist  for  a  verbal 
interview  with  each  project 

*  Conduct  interviews  with  each  project 
(2-3  times  per  year) 

*Potter,  N.,  Sakry,  M.,  — Mfcing  Process  Improvement  Work  -  A  Concise  Action 
Software  Managers  and  Practitioners,”  Appendix  F.  Addison-Wesley,  2002. 


Hide  for 
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Mini-assessment  Process 


•  Plan  the  assessment 

•  Meet  with  interviewees  to  explain  what  will  be 
checked,  how  and  why  (answer  questions  and 
concerns) 

•  Perform  the  mini-assessment  (interviewing 
with  questionnaire) 

•  Communicate  the  results  (organization 
summary) 

•  Debrief  the  mini-assessment  process  to 
obtain  feedback  and  buy-in  from  the 
organization 

•  Improve  the  questionnaire  (emphasize  intent 
and  remove  ambiguity) 

•  Take  corrective  action 
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Mini-assessment  Questions 


Sample  mini-assessment  questions 

*  Describe  how  your  team 

-  Performs  inspections  or  walkthroughs  for  key  work 
products  (such  as  code,  design,  test  cases,  and  test 
plans) 

-  Performs  black-box  testing 

-  Performs  white-box  testing 

-  Performs  version  control  of  all  significant  work  products 
(from  plans  to  code) 

*  Do  you  have  adequate  computer  network  stability 
(compared  with  the  problem  reported  in  the  last 
assessment)? 
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Mini-assessment  Findings 


Project  A:  Strengths  and  areas  for  improvement 

Strengths 

-  Inspections  are  performed  on  requirements  and  code. 

-  Black-box  testing  is  performed  against  the  requirements. 

-  White-box  testing  is  performed  on  critical  code. 

-  Work  products  are  under  configuration  management  (in  other  words, 
project  plans,  requirements,  code,  test  plans,  and  test  cases). 

Areas  for  improvement 

-  Computer  network  stability  has  not  changed  since  it  was  reported  in 
the  last  assessment. 

-  Project  plans  for  projects  larger  than  three  months  would  benefit  from 
inspection. 

-  Test  plans  would  benefit  from  inspection  to  reduce  the  amount  of 
redundancy  in  the  test  approach. 
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Example  Mini-assessment  Data  -i 


Improvement  Progress 
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Observations  and  Corrective  Actions  -i 

*  Little  progress 

*  Will  likely  be  Level  2  by  second-half  of  year  6 

*  Corrective  actions: 

-Report  mini-assessment  data  (overall  score  and  trend) 
to  CEO  and  division  heads  -  get  executive  visibility 

-  Replan  improvement  effort  for  each  project  (goals  and 
problems) 

-Task  new  engineering  manager  with  pulling  Level  2 
practices  into  each  project  using  bi-weekly  project 
reviews  and  EPG  assistance 
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%Total 

criteria 

adopted. 


Yr  1  Yr  1  Yr  1  Yr2  Yr2  Yr2  Yr3  Yr3 
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Observations  and  Corrective  Actions  -2 

*  Progress  is  good 

*  Large  jump  between  May  and  Sept  (year  2)  due  to 
adoption  of  change  management  process,  build 
process,  and  process  assurance 

*  Corrective  actions: 

-Adopt  remaining  Level  2  practices  based  on  current 
project  problems 
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Rules  of  Thumb  for  Measuring  Adoption  -i 


•  Always  be  careful  when  making  measurements: 

-what  you  measure  might  change  when  you  measure  it 

-always  keep  in  mind  the  quality  of  the  answer;  a  YES 
should  be  followed  with: 

»  how  well  does  that  work? 

»  does  it  help  you? 

»  how  often  do  you  do  that? 

»  do  you  do  this  in  a  crisis? 
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Rules  of  Thumb  for  Measuring  Adoption  -2 

•  Use  balanced  metrics  -  don’t  just  have  a  goal: 

- Adpt  50  percent  of  all  Level  2  practices  by  December” 

•  It  may  be  tempting  for  a  project  team  to  implement  the 
easiest  50  percent  of  the  elements  in  the  framework 

•  Balance  this  with  a  metric  that  tracks  progress  toward  a 
business  goal,  e.g., 

- Redce  defects  reported  from  the  field  by  30  percent” 

- Ensur$)roduct  deliveries  are  no  more  than  15  percent  late” 


Tracking  and  observing  multiple  indicators  show  whether 
progress  is  being  made  on  issues  that  impact  the  business 
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Rules  of  Thumb  for  Measuring  Adoption  -3 


•  The  goal  is  to  establish  whether  the  practice  is  a 
common  and  beneficial  behavior  of  the  project  team 

*  Measurements  take  time  to  become  refined,  accurate, 
useful  and  effective 

*  Measurement  results  MUST  be  treated  with  care  -  don’t 
attribute  project  names 

•  The  intent  of  the  measure  must  be  explicitly  stated  to 
explain  how  the  results  will  be  used 
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-Are  We  Making  Progress  on  the  Goals? 

-Are  We  Making  Progress  on  Our  Improvement  Plan? 

-Are  We  Making  Progress  on  the  Improvement 
Framework? 

-What  Lessons  Have  We  Learned  So  Far? 
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What  Lessons  Have  we  Learned  so  Far? 

*  Invite  people  who  are  willing  to  be  frank  and  candid 

-  e.g.,  PI  users,  skeptics,  managers 

*  Select  a  good  objective  facilitator 

*  Two  hours  or  less  to  avoid  team  fatigue 

Lessons  learned  agenda 

1.  Clarify  the  scope  of  the  session  [io  mins] 

2.  Determine  strengths  (what  went  well)  [20  mins] 

3.  Determine  areas  for  improvement  [30  mins] 

4.  Set  priorities  [30  mins] 

5.  Determine  corrective  actions  [30  mins] 

1 .  Where  to  use  the  lesson 

2.  Specific  corrective  actions 
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Lesson 

Where  to  Use 

Lesson 

Decentralizing  the  action  plan  gives  each  project  team 
ownership  over  its  plan. 

Corrective  action  (CA)  =  Continue  having  three  separate  action 
plans,  one  for  each  of  the  three  product  lines. 

Planning 

Don’t  preach  when  an  example  can  say  everything  for  you. 

CA=  Have  one  project  each  month  conduct  a  one-hour  briefing 
describing  the  use  and  benefits  of  a  new  technique. 

Implementing 

Guide  people  in  applying  each  new  technique  to  their  work. 
People  have  so  much  going  on  that  they  do  not  know  where  to 
start. 

Implementing 

CA=  For  each  process  in  the  process  assets  library  (PAL),  add 
tailoring  guidelines  to  explain  when  the  process  should  be  used. 
Provide  one-on-one  coaching  to  new  project  teams. 
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Lessons  Learned  -  Improvement  Areas 


The  process-centric  approach  was  very  difficult  to  sell. 

CA  =  adopt  the  goal-problem  approach. 

Planning 

Using  the  same  communication  technique  as  everyone  else 
allows  the  message  to  be  lost. 

CA=  use  bright  pink  8.5  x  11 -inch  cards  &  pizza  lunches. 

Implementing 

Allowing  private  data  to  become  public  sets  perilous 
expectations. 

CA  =  brief  management  on  new  metrics  policy. 

Planning 

Be  careful  of  what  information  you  ask  for!  [Process  Assets 
Library] 

CA  =  stop  measuring  the  %  of  projects  that  submit  to  the  PAL. 

Clean  out  the  PAL. 

Planning 

Using  a  scoring  system  for  process  adoption  can  encourage 
inappropriate  behavior. 

CA  =  stop  measuring  #inspections/year.  Re-look  at  all  metrics  that 
can  be  optimized  but  lead  to  little  benefit. 

Checking 
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*  Measure  what  you  care  about 

*  Practice  measuring 

*  Lessons-learned  data  provides  additional  feedback 

*  Take  corrective  action  based  on  what  you  learn 
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Exercise:  Checking  Progress 

Result: 

1.  Pick  1-3  goals  and  use  the  GQM 
approach  to  determine  the  metrics  you 
need  to  track  progress  toward  this  goal 

2.  If  your  improvement  program  is  already 
well  underway,  develop  a  plan  to 
conduct  a  mini-assessment  for  one  or 
more  projects  to  assess  current 
progress  and  next  steps 

3.  Discuss  lessons  learned  regarding  your 
improvement  program  with  some  of  the 
program’s  participants 


Action  Items 
1. - 

2. - 


3. 
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Blank  slide 
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Engineering  Process  Group 


A  means  of  implementing  a  successful 
process  improvement  program 


A  Engineering  Process  Group  is  a  group  chartered  to  facilitate 
engineering  process  improvement  within  an  organization. 

It  helps  the  organization  determine  areas  for  improvement,  plan  the 
improvement  effort  and  implement  it. _ 


1 
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Technical  K 
working  group 


EPG  is  a  focal  point  for  working  groups 
and  helps  targets  adopt  new  practices 
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*  Not  part  of  QA.  Not  tied  to  any  one  project. 

*  Geographical  and  cultural  dispersion. 

*  Career  path:  may  be  rotational  (2  years),  good  for  project  leaders  and 
managers. 

*  Must  be  seen  as  a  respected  support  function  to  the  projects. 

*  Champions  often  become  EPG  members.  Others  must  be  recruited. 
Do  not  accept  the  unwanted,  idle,  or  disliked! 
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EPG  Roles 


Presenter  /  teacher 


Coach 


Facilitator 


Politician  /  go-between 
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Relation  of  an  EPG  to  QA 

*  Perceived  as  helping  the  practitioners  and  managers 
adopt  new  practices 

*  Not  QA  (Quality  Assurance) 
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Focal  Point  for  Improvement  and 
Conveyer  of  Information 


Lessons 

learned 


Models  /  standards 


Sjii/ 


Study  /  reading 


Deploys  new  concepts 
in  a  usable  form 
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Skills 

Must 

Staffing  Your  EPG 

Intense  interest  in  process  improvement 

Available  full-time  for  EPG  work  (or  dedicated  part-time) 

Experience  of  and  project  management  problems 

Able  to  work  well  with  people 

Persuasive,  resourceful  (will  try  many  different  things) 

Attitude  of  helping  others  and  good  rapport  with  organization 
Communication  skills  (able  to  make  all  concepts  practical) 

Nice 

Good  coach  and  quality  advocate 

Sensory  acuity  (able  to  adjust  approach  based  on  results) 

Wish 

Knowledge  of  process  improvement,  SEI,  Deming,  Crosby 

Training 

e.g.,  CMMI,  process  appraisals,  consulting,  engineering  and 
management  skills,  metrics 
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Typical  Things  that  an  EPG  Does 

*  Helps  analyze  expectations  versus  current  practices 

*  Conducts  engineering  process  assessments 

*  Helps  write  plans  for  improvement 

*  Helps  implement  the  plans  -  facilitates  improvement 

*  Maintains  sponsorship 

*  Acts  as  an  information  source,  sharing  information 
across  the  organization 

*  Promotes  and  teaches  new  behaviors  (process 
consultation  and  technology  insertion  focal  point) 

*  Helps  write  processes  and  work  aids 

*  Measures  process  improvement 
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Things  that  an  EPG  Does  Not  Do 


*  Reveal  specific  confidential  information  to 
management  or  others 

*  Take  sole  responsibility  for  the  actual  improvement 

*  Write  processes  or  procedures  in  a  vacuum 

*  Issue  edicts  or  strong-arm  changes 


©  Copyright  2002-2007  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  2.3 


162 


THE 

PRC 

GRC _ 

EPG  Steering  Committee 

•  Select  key  sponsors,  champions  and  managers  to 
ensure  buy-in 

•  Explain  issues,  ideas,  practices 

•  Review  the  status  of  improvement  efforts 

•  Tell  them  what  they  need  to  do  to  help 

•  Tell  them  what  resources  you  need 

•  Ask  them  to  help  resolve  issues 

•  Obtain  support  for  additional  staffing 

•  Include  targets  with  success  stories 

•  Include  related  efforts  such  as  quality  teams  and 
working  groups  (credit  and  praise) 
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Tutorial  Objectives 


•  Emphasize  the  importance  of  identifying  and  managing  program  risks  and 
issues 

•  Clarify  the  difference  between  risks  and  issues 

•  Present  Risk/Issue  Avoidance 

-  The  elimination  of  the  sources  of  high  risk  by  replacing  them  with  a  lower-risk 
alternatives 

-  The  establishment  of  sound  technical,  programmatic  and  management 
practices  and  activities  early  and  their  execution  throughout  the  entire  life 
cycle  to  reduce  risks  and  issues 

•  Present  opportunities  associated  with  risks/issues 

—  When  perusing  new  opportunities  risks  and  issues  are  encountered 

•  By  taking  calculated  risks  organizations  may  realize  future  opportunities 

-  Opportunities  to  improve  program  and  project  performance  may  surface  while 
resolving  risks  and  issues 
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Where  Are  We 


•  Tutorial  Objectives 

^  •  Introduction 

-  Definitions 

•  Reasons  for  Risk/Issue  Management 

•  Opportunities 

•  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Opportunity/Risk/Issue/Opportunity  Scenario 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Introduction 

Definitions 

•  Risks  (IEEE  Std  1540-2004;  Standard  for  Software  Life  Cycle  Processes) 

-  Program  and  project  risks  are  the  likelihood  of  an  event,  hazard,  threat,  or 
situation  occurring  and  its  undesirable  consequences 

•  Risk  (Project  Management  Body  of  Knowledge  PMBOK) 

-  An  uncertain  even  or  condition  that,  if  it  occurs,  has  a  positive  or  negative  effect 
on  project's  objectives 

•  Issues  (QATAR  National  Project  Management) 

-  An  issue  is  something  currently  happening  that  is  having  a  negative  impact  on  the 
project  and  requires  resolution  for  the  project  to  proceed  successful 

•  Issues 

—  An  issue  can  be  associated  with  a  risk  if  the  risk  is  realized;  has  occurred 

•  Opportunity  (The  American  Heritage  Dictionary) 

-  A  favorable  or  advantageous  combination  of  circumstances 

-  A  chance  for  progress  or  advancement 

•  Opportunity  (PMBOK) 

-  A  condition  or  situation  favorable  to  the  project,  a  positive  set  of  circumstances,  a 
positive  set  of  events,  a  risk  that  will  have  a  positive  impact  on  project  objectives, 
or  a  possibility  for  positive  chances 
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Introduction 

Definitions 


•  Risk  Response 

-  The  process  of  developing  options  and  actions  to  enhance 
opportunities  and  reduce  threats  to  project  objectives  PMBOK 

-  Includes  Mitigation  and  Contingencies 

-  Includes  acceptance  of  the  risk  or  issue  consequence 

•  Mitigation 

-  Risk  mitigation  implies  an  elimination  or  reduction  in  the  probability 
of  risk  occurrence  PMBOK 


•  Contingency 

—  Issue  contingency  implies  an  elimination  or  reduction  of  the  impact  of 
issues  or  alternative  actions  taken 
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Introduction 

•  We  will  start  with  Opportunities  and  end  with  Opportunities 

—  Opportunity 
—  Risk 
—  Issue 
—  Opportunity 
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*  Managing  Risks ,  Methods  for  Software  Systems  Development;  Dr.  Elaine  M.  Hall,  SEI  Series  in  Software  Engineering 
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Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

^  •  Reasons  for  Risk/Issue  Management 

•  Opportunities 

•  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Questions/Discussion 

•  References 

•  Contact  Information 
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Reasons  for  Risk/Issue  Management 


•  When  developing,  delivering,  and  acquiring  systems  and 
products 

-  developers  and  acquirers  face  many  challenges 

•  Challenges  can  exist  with  many  items  and  activities: 

-  Cost 

-  Schedule 

-  Technical 

-  Management 

-  Programmatic 

-  Process 

-  Performance 

-  Others? 
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Reasons  for  Risk/Issue  Management 


•  Consequences  may  be  numerous  if  challenges  are  not  mitigated 

-  Cost  overruns 

-  Late  deliveries 

-  Technically  inadequate 

-  Programmatic  difficulties 

-  Irate  management 

-  Irate  customer 

-  Canceled  project 

-  Loss  of  market  share 

-  Missed  opportunities 

-  Others? 
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Reasons  for  Risk/Issue  Management 

•  There  are  solutions  for  an  organization  to  help  mitigate  these 
challenges 

-  Proper  program/project  management 

-  Proper  program/project  planning 

-  Program/project  monitoring  and  control 

-  Adequate  budgets 

-  Adequate  schedules 

-  Proper  requirements  development  and  management 

-  Contract  tracking  and  oversight 

-  Product  evaluation 

-  Performance  management 

-  Risk/Issue  management 

-  Quality  assurance 

-  Configuration  management 

—  Independent  Verification  and  Validation  (IV&V) 

-  Others? 
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Reasons  for  Risk/Issue  Management 


•  It  is  important  to  recognize  risks  and  issues  early  and  manage 
them  to  reduce  or  eliminate  their  impact  if  they  occur 

•  Often  organizations  neglect  risk  and  issue  management  or  do 
not  provide  sufficient  attention  to  them 
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Compliance  with  CMMI  ® 


•  Software  Engineering  Institute  (SEI)  Capability  Maturity 
Model  Integration  (CMMI)  _ 


-  CMMI  for  Development  vl. 3 

-  CMMI  for  Acquisition  vl.3 

-  CMMI  for  Service  vl.3 


r 


v 


All  have 

Risk  Management  and 
*lssue  Management 
Process  Areas 


A 


J 


In  order  for  organizations  to  be  compliant  with  CMMI  they  need  to 
establish  risk  and  issue  management  capabilities 


*  Issue  management  is  implied  in  the  Project  Planning,  Project  Monitoring  and 
Control,  Quality  Assurance  and  Configuration  Management 
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Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

•  Reasons  for  Risk/Issue  Management 
■=>  •  Opportunities 

•  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Take  Advantage  of  Opportunities 

•  Risks  are  not  always  undesirable  events 

-  Taking  risks  can  sometimes  be  necessary 

-  If  we  are  not  willing  to  take  calculated  risks  our  advancement  in 
technology  and  business  may  be  hindered 

•  We  have  to  be  circumspect  with  the  risks  we  are  willing  to 
take 

—  And  MANAGE  them  properly 


Need  to  balance  Opportunities  with  Risks 
and  Risks  with  Opportunities 
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Take  Advantage  of  Opportunities 


Opportunities/Risks 

*Running  away  from  risk  is  a  no-win  strategy.  Unless  your 
organization  has  been  sound  asleep  for  the  past  30  years, 
all  the  relatively  risk-free  opportunities  have  long  since 
been  exploited.  The  remaining  high-opportunity  areas 
are  rife  with  risk.  It  is  in  these  areas  and  these  alone 
where  you  need  to  focus  your  attention,  skills  and 
resources. 


*  Managing  Risks >  Methods  for  Software  Systems  Development;  Dr.  Elaine  M.  Hall, 

SEI  Series  in  Software  Engineering 
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Ice  Cream  Opportunity 
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Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

•  Reasons  for  Risk/Issue  Management 

•  Opportunities 

i=>  •  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Risk  Management  Process 


•  Risk  Management  is  an  overarching  process  that  encompasses 

-  Risk  Planning 

-  Risk  Identification 

-  Risk  Analysis 

-  Risk  Response 

-  Risk  Monitoring  and  Control 

PMBOK 
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Risk  Management  Flow 


Florence 


Risk  Management  Planning 

•  Risk  management  planning  is  the  process  of  deciding  how  to 
approach  and  conduct  the  risk  management  activities  for  a  project 

•  Planning  is  important  to 

-  Ensure  the  level,  type  and  visibility  of  risk  management  are  commensurate 
with  both  the  risk  and  importance  of  the  project  to  the  organization 

-  Provide  sufficient  resources  and  time  for  risk  management  activities 

-  Establish  an  agreed-upon  basis  for  evaluating  risks 

•  Risk  planning  should  be  completed  early  during  project  planning 
PMBOK 
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Risk  Management  Team 

•  The  Risk  Managment  planning  activity  may  assign  a  Risk 
Management  Team  to  administer  the  Risk  Management  Program 

•  A  Risk  Manager  may  be  assigned  to  manage  the  Risk  Management 
Team 

•  A  Risk  Management  Board  may  be  chartered  to  review,  accept, 
decline,  transfer  and  escalate  risks 

•  Hierarchy  Governance  Boards  may  exist  for  escalation  of  risks  based 
on  thresholds 

•  Everyone  on  the  program/project  is  responsible  for  risk 
management 


The  level  of  this  implementation  depends  on  the  size,  scope, 
critically,  safety,  security,  etc.  of  the  application 
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Risk  Management  Plan 

•  Risk  management  planning  needs  to  be  part  of  project  planning 

•  A  risk  management  plan  can  be  a  stand  alone  plan  or  part  of  the  project 
plan 

•  The  risk  management  plan  needs  to  be  tailored  to  the  scope  of  the 
application 

•  The  concepts  provided  in  this  tutorial  can  be  used  to  develop  the  plan 
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Risk  Management  Plan  Outline 


•  Introduction 

•  Project  Description 

•  Risks/Issue/Opportunity 
Descriptions 

•  Risk  Identification 

•  Risk  Analysis 

•  Risk  Response 

-  Risk  Acceptance 

-  Risk  Avoidance 

-  Risk  Transfer 

-  Risk  Escalation 

-  Risk  Mitigation _ 


Risk  Monitor  and  Control 
Risk  Register 
Issue  Management 
Issue  Contingency 
Risk/Issue  Opportunities 
Risk/Issue  Training 
Glossary 
References 
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Risk  Management  Artifacts 

•  Risk  Management  is  supported  with: 

—  Risk  Management  Policy 

-  Risk  Management  Charter 

-  Risk  Process  Description 

-  Risk  Management  Plan 

-  Risk  Management  Procedures 

-  Risk  Management  Guidelines 

-  Risk  Management  Training 

•  This  presentation  will  not  dwell  on  these  but  content  of  this 
presentation  can  support  their  construction/implementation 


Z\£<jTT&^ &/  Florence 


25 


Risk  Identification 


•  Risk  Identification  is  the  activity  that: 

-  Identifies  potential  and  current  risks 

-  Examine  elements  of  the  program  to  identify  associated 
potential  root  causes  of  risks 

-  Begin  their  documentation 

-  Sets  the  stage  for  their  successful  management 

-  Risk  identification  begins  as  early  as  possible  in  successful 
programs  and  continues  throughout  the  life  of  the  program 

-  Project  and  stakeholders  from  outside  and  inside  the  project 
should  be  involved  in  risk  identification 
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Risk  Identification 


•  Risk  can  be  associated  with  all  aspects  of  a  program;  e.g. 

-  Requirements 

-  Threat 

-  Security 

-  Technology  maturity 

-  Supplier  capability 

-  Design 

-  Schedule 

-  Cost 

-  Performance 

-  Etc. 
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Risk  Description 

•  As  risks  are  identified  it  is  important  to  correctly  describe  them 

•  A  well-written  risk  statement  contains  three  main  components: 

-  Cause  -  The  negative  conditions  that  currently  exist  relative  to  the  risk 

•  Identification  of  root  cause(s)  of  the  risk 

•  This  provides  justification  that  a  risk  exists 

-  Probability  of  Occurrence  -  The  likelihood  of  the  occurrence  of  the  risk 

•  Within  a  future  time  frame 

•  Ora  future  event 

-  Consequence  -  The  effect(s),  negative  impact(s)  to  the  program(s)  in  case 
the  risk  occurs 

•  The  consequence  should  be  related  to  at  least  cost,  schedule,  scope  and 
performance 

•  Consequence  could  also  result  in  opportunities  that  may  surface  in  correcting 
the  problems 
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Risk  Description 


•  The  risk  is  written  in  a  chain  of:  Cause:  IF;  THEN 

Example 

An  Interface  Working  Group  has  not  been  formed  and  a  plan  to  form  one 
does  not  exist. 

IF  key  stakeholders  cannot  agree  on  interface  protocol  by  11/14/2011; 

THEN  the  schedule  for  development  and  delivery  will  be  delayed  causing 
cost  overruns. 


NOTE:  The  cause  includes  assurance  that  the  reason  for  the  risk  is  valid.  I.e.,  is 
there  a  compelling  reasons(a  root  cause)  to  assume  that  stakeholders  cannot 
agree  on  the  interface  protocol  by  11/14/2011?  Not  just  pie  in  the  sky. 
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Risk  Description 

•  Proper  risk  descriptions  helps  manage  the  right  risks 

-  Risk  management  is  time  and  resource  consuming 
—  Managing  "non-risks"  is  not  cost  effective 

•  Example 

-  A  risk  may  be  identified  as  a  risk  that  component  YYY  will  be  provided 
late 

-  Writing  this  risk  as: 

•  IF  component  YYY  is  delivered  late;  THEN  ... 

-  May  fail  to  inspire  interest  and  action 

•  The  risk  is  too  vague,  or 

•  There  is  no  clear  reason  why  this  is  a  risk 

-  In  this  case  one  needs  to  identify  causal  conditions  that  may  prevent 
timely  delivery  of  YYY  If  there  are  none  this  is  not  a  risk! 
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Risk  Description 

•  The  cause  may  be  included  in  the  "IF"  statements  for  some  risks 


IF  scheduled  delivery  of  component  YYY  continues  to  slip  beyond  11/14/2011; 
THEN  system  integration  will  also  slip  causing  the  system  to  incur  cost  and 
schedule  overruns. 


•  Cause 

-  Written  in  the  IF  statement 

-  Schedule  continues  to  slip 

•  Occurrence 

-  Schedule  delivery  slips  beyond  11/14/2011 

•  Consequence 

-  Cost  and  schedule  overruns 

Florence 
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Risk  Description 

•  Avoid  writing  the  mitigation  strategy  into  the  risk  description 

-  A  mitigation  strategy  is  developed  after  the  risk  has  been  approved 
and  analyzed 

Examples 

Requirements  have  always  been  a  problem  in  passed  projects  within  this 
organization.  IF  requirements  are  not  reviewed  and  verified;  THEN 
requirement  defects  will  migrate  into  the  design. 

•  Reviewed  and  verified  are  possible  mitigation  strategies 

•  Write  the  risk  in  a  chain  of  cause,  occurrence,  consequence 

Requirements  have  always  been  a  problem  in  passed  projects  within  this 
organization.  IF  defective  requirements  are  not  discovered  and  corrected  by 
PDR;  THEN  requirements  defects  will  migrate  into  the  design  and 
implementation  causing  rework,  and  cost  and  schedule  impacts. 
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Risk  Description 


•  Risks  must  be  written  in  a  clear,  concise  and  unambiguous 
fashion 

•  Words  and  phrases  that  may  have  confusing  and  multiple 
interpretations  must  be  avoided 
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Ambiguous  Words 

•  Avoid  ambiguous  words  in  describing  risks,  some  examples: 


-  Adequate 

-  Limited 

-  Ad  hoc 

-  Near  real  time 

-  All 

-  Periodic 

-  Always 

-  Portable 

-  Appropriate 

-  Rapid 

-  Clearly 

-  Several 

-  Easy 

-  Slow 

-  Existing 

-  Small 

-  Fast 

-  Sometimes 

-  Flexible 

-  State  of  the  art 

-  Future 

-  Sufficient 

-  If  required 

-  Usable 

-  Immediately 

-  User-friendly 

-  Large 

-  Weight 

-  Light 

-  When  required 

-  Others? 

From  IEEE  standards  and  some  preparatory  requirements  management  plans 
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Also:  http://www.ppi-int.eom/newsletter/SvEN-017.php#artide 
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Risk  Analysis 

•  The  risk  is  submitted  to  the  Risk  Management  Board 

•  The  risk  is  accepted  or  declined  by  the  Board 

—  If  declined  rational  is  conveyed  to  the  submitter 

•  If  accepted  the  Risk  Management  Board  assigns: 

-  A  Risk  Analyst  responsible  for  conducting  risk  analysis  on  assigned 
risks 

•  Supported  by  Subject  Matter  Experts  (SMEs) 

—  A  Risk  Owner  responsible  for  ensuring  risks  are  properly  managed 
throughout  their  life 

—  Risk  Analyst  and  Owner  could  be  one  in  the  same 
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Risk  Analysis  Components 

•  Risks  have  the  following  components: 

—  A  future  root  cause(s)  (yet  to  happen)  which 

•  if  eliminated  or  corrected,  would  prevent  a  potential  consequence  from 
occurring 

-  A  probability  of  occurrence  (or  likelihood) 

•  assessed  at  the  present  time  and  updated  when  necessary  of  the  future 
root  cause  occurring 

-  The  consequence  (or  effect/impact)  of  that  future  occurrence 

—  The  time  horizon  during  which  the  consequences  will  occur  if  the  risk  is 
not  mitigated 

—  Risk  Priorities 

•  Mapping  of  probability  of  risk  occurrence  and  risk  consequence 

-  Risk  Triggers 

•  Specific  events  or  conditions  that  indicate  when  to  develop  and  execute 
mitigation  or  contingency  strategies 
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Risk  Analysis 

*  Qualitative  Risk  Analysis 


-  Relative  measure  of  risk  or  asset  value  based  on  ranking  or  separation  into  descriptive 
categories  such  as  low,  medium,  high;  not  important,  important,  very  important;  or  on  a 
scale  from  1  to  10. 

BusinessDictionary.com 

-  An  examination  and  prioritization  of  the  risks  based  on  their  probability  of  occurring  and 
the  impact  on  the  project  if  they  do  occur.  Qualitative  risk  analysis  guides  the  risk 
reaction  process. 

pmpbank.googlepages.com/glossary 


*  Quantitative  Risk  Analysis 

-  Incorporates  numerical  estimates  of  frequency  or  probability  and 

consequence.  In  practice,  a  sophisticated  analysis  of  risk  requires  extensive 
data  which  are  expensive  to  acquire  or  often  unavailable.  Fortunately,  few 
decisions  require  sophisticated  quantification  of  both  frequency  and 
consequences 

—  Shortly  spoken  one  might  say  that  "quantitative  risk  analysis  breaks  down  risks 
from  a  high  medium  low  ranking  to  actual  numerical  values  and  probabilities 
of  occurrence"  for  being  able  to  compute  the  overall  effects 

(comp.  CR0SSWIND7.  p.  423) 
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Root  Causes 


•  A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a 
risk 

•  The  cause  of  the  risk  has  to  be  isolated  and  defined 

—  Root  causes  should  be  initially  identified  when  risks  are  identified 

-  Once  initial  root  cause  are  identified  they  may  need  to  be  analyzed 
further  to  determine  the  actual  deep  rooted  causes  of  the  risks 

-  Root  causes  are  documented  and  they  support: 

•  Establishing  risk  mitigation  and  contingency  strategies 

•  Improvement  opportunities 

•  Root  causes  can  also  be  referred  as  risk  drivers 


Root  Cause  Analysis.  An  analytical  technique  used  to  determine  the  basic 
underlying  reason  that  causes  a  variance  or  a  defect  or  a  risk.  A  root  cause  may 
underlie  more  than  one  variance  or  defect  or  risk.  ( (PMBOK®  Guide)  --  Fourth 
Edition)  Syn:  root-cause  analysis 
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Root  Causes 


Typical  root  causes  may  be  associated  with: 


Threat 

Requirements 
Technical  Baseline 
Test  and  Evaluation 
Modeling  and  Simulation 
Technology 
Logistics 


Management 
Schedules 
External  Factors 
Budget 

Earned  Value  Management 

Production 

Industrial  Capabilities 

Cost 

Others? 
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Root  Causes 

Background  Information 

•  Threat  -  The  sensitivity  of  the  program  to  uncertainty  in  the  threat 
description,  the  degree  to  which  the  program  would  have  to  change 
if  the  threat's  parameters  change 

•  Requirements  -  The  sensitivity  of  the  program  to  uncertainty  in  the 
system  requirements 

•  Technical  Baseline  -  The  approved  and  fixed  configuration  of  a 
technical  item  at  a  specific  time  in  its  lifecycle  that  serves  as  a 
reference  point  for  change  control 

•  Test  and  Evaluation  -  The  adequacy  and  capability  of  the  test  and 
evaluation  program  to  assess  attainment  of  significant  performance 
specifications  and  determine  whether  the  system  is  operationally 
effective,  operationally  suitable,  and  interoperable  with  the  system 
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Root  Causes 

Background  Information 


•  Modeling  and  Simulation  -  The  adequacy  and  capability  of  M&S  to 
support  all  life-cycle  phases  of  a  program  using  verified,  validated,  and 
accredited  models  and  simulations 

•  Technology  -  The  degree  to  which  the  technology  proposed  for  the 
program  has  demonstrated  sufficient  maturity  to  be  realistically 
capable  of  meeting  all  of  the  program's  objectives 

•  Logistics  -  The  ability  of  the  system  configuration  and  associated 
documentation  to  achieve  the  program's  logistics  objectives  based  on 
the  system  design,  maintenance  concept,  support  system  design,  and 
availability  of  support  data  and  resources 
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Root  Causes 

Background  Information 


•  Management  -  The  degree  to  which  program  plans  and  strategies  exist 
and  are  realistic  and  consistent.  The  program  support  team  should  be 
qualified  and  sufficiently  staffed  to  manage  the  program 

•  Schedule  -  The  sufficiency  of  the  time  allocated  for  performing  the 
defined  tasks 

•  External  Factors  -  The  availability  of  resources  external  to  the  program 
that  are  required  to  support  the  program  such  as  facilities,  resources, 
personnel,  government  furnished  equipment,  etc. 

•  Budget  -  The  sensitivity  of  the  program  to  budget  variations  and 
reductions  and  the  resultant  program  turbulence 

•  Earned  Value  Management  (EVM)  -  The  adequacy  of  the  EVM  process 
and  the  realism  of  the  integrated  baseline  for  managing  the  program 
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Root  Causes 

Background  Information 


•  Production  -  The  ability  of  the  system  configuration  to  achieve  the 
program's  production  objectives  based  on  the  system  design, 
manufacturing  processes  chosen,  and  availability  of  manufacturing 
resources 

•  Industrial  Capabilities  -  The  abilities,  experience,  resources,  and 
knowledge  of  the  contractors  to  design,  develop,  manufacture,  and 
support  the  system 

•  Cost  -  The  ability  of  the  system  to  achieve  the  program's  life-cycle  cost 
objectives.  This  includes  the  effects  of  budget  and  affordability 
decisions  and  the  effects  of  inherent  errors  in  the  cost 
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Probability  of  Occurrence 

•  Probability  of  occurrence  assessed,  at  the  present  time,  is  the 
probability  of  a  future  root  cause  occurring 

•  The  chance  of  a  risk  occurring  is  rated  on  a  scale  between  >0 
and  1 

•  When  the  probability  of  occurrence  =  1;  (100%) 

-  The  risk  has  occurred;  it  then  becomes  an  issue  and  is  managed  as  an 
issue 

•  For  most  risks,  estimating  the  precise  probability  of 
occurrence  may  be  difficult 

-  Analysis  by  SMEs  may  be  necessary,  and  often  using  Best  Engineering 
Judgment 
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Probability  Scores 

•  Probability  of  occurrence  may  begin  with  a  qualitative 

description  of  probability,  which  will  tie  to  a  numeric  range  of 
probability. 


Sample  Risk  Probability  Scores 


Probability  Description 

Probability  %  of 
Occurrence 

Very  High  (Extremely  likely) 

>81%  and  =100% 

High  (Probable) 

61% -80% 

Medium  (Possible) 

41% -60% 

Low  (Unlikely) 

21% -40% 

Very  Low  (Highly  improbable) 

>1%  -  <20% 
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Consequence  of  Risk  Occurrence 

(Impact) 

•  Risks  are  reviewed  for  the  effect  that  they  would  have  on  the 
project's  objectives  and  other  elements  of  the  program 

•  The  level  of  impact,  may  be  rated  from  very  low  (1)  to  very 
high  (5),  and  is  assessed  against  at  least  four  categories: 

-  Cost 

-  Schedule 

-  Scope 

-  Performance 
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Consequence  of  Risk  Occurrence 


Program/Project 

Objective 

Very  Low 

Minor 

Low 

Moderate 

Medium 

Serious 

High 

Critical 

Very  High 

Catastrophic 

Cost 

Insignificant 

increase 

Increase 

<  2%  of 

budget  baseline 

Increase 

2-5%  of 
budget  baseline 

Increase 

6-10%  of 
budget  baseline 

Increase 

>  10%  of 
budget  baseline 

Schedule 

Insignificant 

slippage 

Slippage  <  2%  of 
project  baseline 
schedule 

Slippage  2-5%  of 
project  baseline 
schedule 

Slippage  6-10% 
of  project 
baseline 

schedule 

Slippage  >  10% 
of  project 
baseline 

schedule 

—  OR  — 

Slippage  past  a 
milestone 
mandated  by 
Congress 

Scope 

Scope  decrease 
barely  noticeable 

Minor  areas  of 
scope  affected 

Major  areas  of 
scope  affected 

Scope  reduction 
unacceptable  to 
sponsor 

Project  outcome 
is  effectively 
useless 

Performance 

Performance 
degradation 
barely  noticeable 

Performance 
degradation 
noticeable,  but 
does  not  fail 
acceptance 
criteria 

Performance 

reduction 
requires  sponsor 
approval 

Performance 

reduction 
unacceptable  to 
sponsor 

Project  outcome 
is  effectively 
useless 
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Time  Horizon 


•  There  are  at  least  three  dates  that  may  be  specified  for  each 
risk: 

—  Near  risks  are  those  in  which  the  earliest  date  of  the  risk  impact  is 
within  xx  days  of  the  present  date 

—  Mid  risks  are  those  in  which  the  earliest  date  of  risk  impact  is  between 
xx  and  zz  days  from  the  present  date 

-  Far  risks  are  those  in  which  the  earliest  dates  of  the  risk  impact  are 
greater  than  zz  days  from  the  present  date 

(xx<zz) 

•  These  dates  are  used  as  triggers  to  track  when  the  risk  will 
begin  to  impact  the  program  and/or  when  the  risk  has  been 
overcome  by  events 


Near  Risks 


Mid  Risks 


Far  Risks 


XX 


ZZ 


Time 
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Risk  Exposure 

•  Risk  exposure.  (ISO/IEC  16085:2006  Systems  and  software 
engineering--Life  cycle  processes--Risk  management 

(1)  the  potential  loss  presented  to  an  individual,  project,  or 
organization  by  a  risk 

(2)  a  function  of  the  likelihood  that  the  risk  will  occur  and  the 
magnitude  of  the  consequences  of  its  occurrence 

•  Risk  exposure  can  also  be  called  Risk  Priority 

-  The  priority  of  a  risk  helps  to  determine  the  amount  of  resources  and  time 
that  should  be  dedicated  to  managing  and  monitoring  the  risk 

-  Very  Low,  Low,  Medium,  High,  and  Very  High  priority  is  assessed  by  using 
probability  and  impact  scores 

-  The  potential  timing  of  a  risk  event  may  also  be  considered  when 
determining  risk  management  actions 
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Risk  Priorities 


Impact 
Very  High 
High 

Medium 

Low 

Very  Low 


Very  Low  Low  Medium  High  Very  High 
Probability  of  Occurrence 


Very  Low  Low 

Priority  Priority 

Florence 


Medium  High  Very  High 

Priority  Priority  Priority 
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Risk  Priority  vs.  Mitigation/Contingency 


Very  Low 

Low 

Medium 

High 

Very  High 

Priority 

Priority 

Priority 

Priority 

Priority 

1 

1 

1 

1 

1 

Risk 

Monitor 

Monitor 

Monitor 

Monitor 

Watch 

List 

Develop 

Develop 

and  execute 

Contingency 

Mitigation 

Strategy 

Strategy 

Very  Low  Priority  Risks  are  placed  in  a  Risk  Watch  List  which  are 

periodically  monitored. 

Other  Risks  are  monitored  more  aggressively. 
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Risk  Priorities  &  Time  Horizon 

The  potential  tinning  of  a  risk  event  may  also  be  considered 
when  determining  risk  management  actions 


Time 

Horizons 


New 

Priorities 


Far 

Near 

Impact 

Impact 

Very  Low 

Medium 

Priority 

Priority 

Medium 


Far  Near 

Impact  Impact 


Low  High 

Priority  Priority 


Far  Near 
Impact  Impact 


Medium  Very  High 
Priority  Priority 


Time  Horizons  may  influence  the  Risk  Priority 

(Mid  Time  Horizon  should  not  influence  risk  priority  change) 

Priorities  may  change  over  time 
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Identifying  Triggers 

•  Triggers  are  specific  events  or  conditions  that  indicate  when 
to  execute  mitigation  or  contingency  strategies 

•  Unless  a  condition  is  immediate,  a  trigger  should  be  defined 

•  Examples  of  triggers  may  include: 

-  Cost  performance 

-  Schedule  performance 

-  Results  of  management  reviews 

-  Occurrence  of  the  risk 

•  as  a  trigger  for  execution  of  contingency  strategies 
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Risk  Response 


•  Risk  response  is  the  process  of  developing  options  and 
determining  actions  to  enhance  opportunities  and  reduce 
threats  to  the  project's  objectives 

•  Risk  response  must  be 

-  Appropriate  to  the  significance  of  the  risk 

-  Cost  effective  in  meeting  the  challenge 

-  Timely  and  realistic  within  the  project  contend 

-  Agreed  to  by  all  parties  involved 
PMBOK 
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Risk  Response 


•  Risk  Responses  has  at  least  five  components 

-  Acceptance 

-  Avoidance 

-  Transfer 

-  Escalate 

-  Mitigate  (contingencies  -for  issues) 

•  Acceptance  -  Accept  the  consequences  of  the  risk  occurring 

-  Other  responses  may  not  be  possible 

-  Cost  to  respond  may  be  greater  than  the  benefit 

-  May  not  be  possible  to  prevent  the  impact  if  the  risk  occurs 

-  Impact  may  be  negligible 

-  Risk  may  be  imminent  and  should  be  handled  as  an  issue 
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Risk  Avoidance/Transfer 

-  Avoidance  (covered  later  in  presentation) 

•  Eliminate  the  sources  of  high  risk  and  replace  them  with  a  lower- 
risk  alternative 

•  Avoid  risks  with  good  management  and  engineering  practices 

-  Transfer  -  Shift  the  responsibility  of  managing  and  resolving  the 
risk  to  another  party 

•  May  be  better  able  to  manage  the  risk 

•  May  be  the  proper  owner  of  the  risk 

•  Transfer  could  be  from  one  party  to  another  within  the  same 
organization 

•  Transfer  could  be  to  a  completely  different  organization 
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Risk  Escalation 

•  Escalation  -  Risks  should  be  managed  at  the  lowest  practical 
level 

-  But  conditions  may  arise  where  a  risk  should  be  escalated  to  higher 
levels  of  management  or  beyond  the  program/project 

-  The  next  higher  organizational  (Governance)  entity  may  be  able  to 
better  to  handle  the  risk/issue 

-  Thresholds  may  exist  that  determine  escalation 

•  Cost  of  impact 

•  Schedule  effect  of  Impact 

•  Scope  of  impact 

•  Performance  effect  of  impact 

•  Time  critical 

•  Cost  critical 
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Risk  Mitigation 


•  Taking  early  action  to  reduce  the  probability  and/or  impact  of 
a  risk  occurring  is  often  more  effective  that  trying  to  repair 
the  damage  after  the  risk  has  occurred 

•  Adapting  less  complex  processes,  conducting  more  tests,  or 
choosing  a  more  stable  supplier  are  examples  of  mitigation 
actions 

PMBOK 
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Risk  Mitigation 


•  The  following  are  important  guidelines  for  effective  risk 
mitigation: 

-  Prepare  detailed  mitigation  strategies  for  all  medium,  high 
and  very  high  risks 

•  With  sufficient  detail  about  what  is  to  be  done,  when,  where, 
and  by  whom 

-  Develop  mitigation  strategies  as  early  as  possible,  allowing 
time  to  address  risks  needing  special  attention  or  action 

•  Helps  reduce  the  chance  of  having  high-priority  risks  appear 
at  the  last  moment  on  the  critical  path 

-  Prepare  contingency  strategies  for  all  high  and  very  high 
priority  risks  and  risks  imminent  to  occur 
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Risk  Mitigation 

Background  Information 

•  Adaptations  of  the  following  strategies  can  be  applied  to  a 
range  of  risks.  This  list  is  intended  merely  as  a  starting  point 
for  thinking  about  risk  mitigation 

-  Multiple  Development  Efforts  -  Create  competing  systems  in  parallel 
that  meet  the  same  scope  and  performance  requirements 

-  Alternative  Design  -  Create  a  backup  design  option  that  uses  a  less 
risky  approach 

-  Trade  Studies  -  Conduct  studies  to  arrive  at  the  least  risky  solution 

-  Early  Prototyping  -  Build  and  test  prototypes  early  in  the  system 
development 

-  Incremental  Development  -  Design  with  the  intent  of  upgrading 
system  parts  in  the  future 
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Risk  Mitigation 

Background  Information 

-  Technology  Maturation  Efforts  -  Normally,  technology  maturation  is  used 
when  the  desired  technology  will  replace  an  existing  technology  that  is 
available  for  use  in  the  system 

-  Robust  Design  -  This  approach,  while  it  could  be  more  costly,  uses 
advanced  design  and  manufacturing  techniques  that  promote  quality 
through  design 

-  Reviews,  Walk-Throughs,  and  Inspections  -  These  three  actions  can  be  used 
to  reduce  the  probability/likelihood  and  potential  consequences/impacts 
of  risks  through  timely  assessment  of  actual  or  planned  events 

-  Design  of  Experiments  -  This  engineering  tool  identifies  critical  design 
factors  that  are  sensitive,  and  therefore  potentially  high-risk,  to  achieve  a 
particular  user  requirement 
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Risk  Mitigation 

Background  Information 

•  Open  Systems  -  Carefully  selected  commercial  specifications  and 
standards,  which  can  result  in  lower  risks 

•  Use  of  Standard  Items/Software  Reuse  -  Use  of  existing  and  proven 
hardware  and  software,  where  applicable,  can  substantially  reduce  risks 

•  Use  of  Mock-Ups  -  The  use  of  mock-ups,  especially  man-machine  interface 
mock-ups,  can  be  used  to  conduct  early  exploration  of  design  options 

•  Modeling/Simulation  -  Modeling  and  simulation  can  be  used  to  investigate 
various  design  options  and  system  requirement  levels 

•  Key  Parameter  Control  Boards  -  The  practice  of  establishing  a  control 
board  for  a  parameter  may  be  appropriate  when  a  particular  feature  (such 
as  system  weight)  is  crucial  to  achieving  the  overall  program  requirements 
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Issue  Contingency 


•  Issue  contingencies  are  developed  for  high  and  very  high  priority 
risks 

•  Covered  further  under  issues 
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Ice  Cream  Risk 
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Risk  Monitoring  and  Control 


•  In  order  to  effectively  monitor  and  control  risks  a  Risk 
Repository  needs  to  be  established 

-  Also  called  a  Risk  Register 

•  There  are  many  risk  tools  that  provide  repository  capabilities: 

-  Home  developed  tools 

-  Commercial  tools 

-  Corporate/agency  tools 


Note:  Risk  register  implementation  may  depend  on  project  size.  A 
month  long  project  might  just  need  a  spread  sheet  table  whereas  a 
multi-year,  geographically  dispersed  project  may  require  an  internet  and 
SQL-based  database  tool. 
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Risk  Control 


•  Risk  control  involves: 

-  Plan  and  execute  actions  that  reduce  the  probability/impact  of  a  risk 
on  the  project's  objectives  (i.e.,  mitigation);  and/or 

-  Establish  a  feasible  impact  control  plan  for  the  realization  of  the  risk 
(i.e.,  contingency) 

-  Control  includes  the  decision  to: 

•  Close  risks 

•  Mark  risks  as  occurred 

•  Update  probability  and  impacts  and  other  details  (conduct  additional 
analysis) 
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Risk  Records 

•  Risk  and  Issue  records  in  a  risk/issue  repository  serve 

several  purposes: 

-  Reporting  and  communicating  information  to  others  who  might 
be  impacted  or  who  might  be  able  to  help  manage  an  item 

-  Providing  risk  lists  and  status  for  reviews  with  stakeholders 

-  Assisting  the  risk  originator  and  owner  the  collection  of 
information  about  an  item 

-  Helping  risk  owners  and  others  to  easily  access  risk  and  issue 
information,  and  manage  that  information  to  the  benefit  of  the 
organization 


Z\£<jTT&^ &/  Florence 


67 


Risk/Issue  Repository 

•  Suggested  contents  of  repository 

-  Category  of  risk  (schedule,  cost,  technical,  management,  etc.) 

-  Risk  Identifier 

•  Unique  alpha  numeric  identifier  of  each  risk/issue 
—  Risk  Name 

•  Descriptive  name  of  risk  or  issue 

-  Risk  Description 

•  Cause 

•  Probability  of  Occurrence  -  IF 

•  Consequence  -  THEN 
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Risk/Issue  Repository 

•  Suggested  contents  of  repository 

-  Stage  in  which  the  risk  exists  at  time 

•  Analysis,  Mitigation,  Contingency,  Issue,  Closed 

-  Priority 

•  Very  Low,  Low,  Medium,  High,  Very  High 

-  Impact 

•  Very  Low,  Low,  Medium,  High,  Very  High 

-  Probability  of  Occurrence 

•  Larger  then  zero  equal  to  1  for  risks 

•  If  1  it  becomes  an  issue 

-  Risk  Triggers 

-  Mitigation  Strategies 

-  Contingency  Strategies 

-  Notes 
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Risk  Monitoring 


•  Risk  monitoring  involves 

-  Tracking  individual  risks,  primarily  by  reviewing  the  status  of  their 
mitigation  strategies,  probability,  consequence,  and  other  information 

—  Risk  Monitoring  includes: 

•  Continuous  reassessment  of  risks 

•  Reporting  on  risks 

-  Continuously  analysis  and  status  updates  in  the  Risk/Issue  Repository 

-  Providing  evidence  to  outside  governance  bodies  that  the  program  and 
its  projects  have  identified  sources  of  uncertainty  and  possible  failure 


Risk  Monitoring  provides  data  and  status  for  Risk  Control  activities 
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Risk  Monitoring 

•  Critical  review  makes  certain  that  the  solution  is  fixing  the 
root  cause  wherever  possible  and  prepares  the  risk  owner 
and  project  to  answer  the  following  questions: 

-  What  is  the  expected  reduction  of  probability  and  impact?  Is  it 
documented?  Is  it  a  sufficient  reduction? 

-  Can  the  option  be  feasibly  implemented  and  still  meet  the  user's 
needs? 

-  Is  time  available  to  develop  and  implement  the  option? 

-  Is  the  mitigation  strategy  affordable  in  terms  of  cost  and  other 
resources  (e.g.,  use  of  critical  materials,  test  facilities,  etc.)?  Does  the 
benefit  outweigh  the  cost? 

-  What  effect  does  the  option  have  on  the  overall  program  schedule? 

-  What  effect  does  the  option  have  on  the  system's  technical  scope  and 
performance? 
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Risk  Monitoring 

•  Risk  reports  can  be  in  the  form  of  text  reports  and/or  graphs 

•  The  following  are  examples  of  reports  and  metrics  that  can  be 
reported: 

—  Open  Risk/Issues  by  Priority  by  Month 
—  Closed  Risks/Issues  Current  Month 
—  Risk/Issue  Plans 

•  Current  and  next  month 
—  Risk/Issue  Transfer  Current  Month 

-  Risk/Issue  Escalation  Current  Month 

-  Risk/Issue  Activity  History 

-  Risk/Issue  for  Management  Attention 
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Risk  Status 


•  Risk  monitoring  includes  constant  monitoring  and  providing 
status  of  the  risks  including 

-  Analysis  of  the  current  status  of  the  risks 

-  Updates  to  the 

•  Probability  of  occurrence 

•  Impact  of  occurrence 

•  Other  risk  parameters 

•  Mitigation  strategies 

•  Contingency  strategies 
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Risk  Closure 


•  Risks  are  closed  when: 

-  They  are  no  longer  a  threat  (the  risk  lessened  or  vanished) 

-  They  have  been  mitigated 

-  They  have  been  transferred  or  escalated 

•  The  new  owners  now  manage  and  monitor  the  risk 

•  When  risks  occur  they  now  become  Issues  and  may  be  closed 
as  risks  or  left  open  in  the  repository  as  Issues 
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Risk 


Probability  of  occurrence  75%  -  High 
Consequence  of  occurrence  -  Very  High 
Risk  Priority  -  High 
Mitigation  -Jump  out  of  car 


Brakes  did  not  work 
Car  stalled  on  tracks 
Battery  dead 


Train  approaching  1/2  mile  away  going  40  mph 
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Enter 


In  Review 

Risk  Management  Flow 


N0I 

Yes 

Reject  Risk 

~T~ 

Close  Risk 

< - 

r lore  nee 


Continuous  Monitor  and  Control  Risk 


Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

•  Reasons  for  Risk/Issue  Management 

•  Opportunities 

•  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Questions/  Discussion 

•  References 

•  Contact  Information 


Z\£<jTT&^ &/  Florence 


77 


Issue  Management 


Used  in 
Next  slide 


An  issue  is  an  event  that  has  occurred  or  will  occur 
with  certainty  and  adversely  affects  the  ability  of  the 
project  or  program  to  meet  its  objectives 

“a)  A  formally  identified  adverse  condition  related  to  a 
program 

al)  Had  not  been  identified  as  a  risk  prior  to  its  occurrence 

a2)  Had  been  identified  as  a  risk  prior  to  its  occurrence  but  had 
not  been  managed  as  a  risk 

_b)  A  program  risk  that  has  occurred  or  is  imminent  to  occur 

•  Reached  its  probability  of  occurrence  of  1  (one)  or  100% 

•  Not  yet  occurred  but  the  probability  of  occurrence  is 
approaching  100% 

-  At  this  time  should  be  managed  as  an  issue 


Some  concepts  from:  PM@UTS  Learning  Program  Step  by  step  guide  to 
project  management 
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Issue  Management 


POW! 


ISSUE 


TIME 


SUPPRISE 


RISK 


POW! 


Risk  mitigation  Strategies 

II - M 


Risk 

occurred 


ISSUE 


100%  probability 
of  occurrence 


Risk  mitigation  Failed 
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Issue  Management 

•  If  program/project  issues  are  not  addressed  they  may: 

—  Affect  program/project  schedule 

—  Increase  program/project  cost 
—  Change  program/project  scope 
—  Diminish  program/project  performance 

•  Issues  can  be  associated  with  all  aspects  of  a  program  (e.g., 
threat,  technology  maturity,  supplier  capability,  design, 
performance  against  plans)  as  these  aspects  relate  across  the 
Work  Breakdown  Structure  and  Integrated  Master  Schedule 
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Issue  Management  Flow 


A 


Florence 


Continuously  Monitor  and  Control  Issue 


Issue  Management 

•  Issue  Management  quickly  identifies  and  effectively  resolves 
problems  as  they  occur  and  involves: 

-  Issue  Identification 

-  Issue  Analysis 

-  Issue  Response 

—  Issue  Monitoring  and  Control 
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Issue  Identification 

•  Issue  identification  activities  are  similar  to  those  for  risk 
identification 

•  If  the  issue  is  one  that  occurred,  without  being  associated 
with  a  risk,  then  it  needs  to  be  identification 

•  Issue  identification  would  have  been  done  if  the  issue 
resulted  from  a  risk  occurring 

—  Regardless  issues  may  need  additional  identification 
•  Especially  in  identifying  root  causes 
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Issue  Description 

•  If  the  issue  was  a  risk  that  occurred  it  should  have  been 
described  with  the  risk  description 

•  If  the  issue  was  not  related  to  a  risk  the  issue  needs  to  be 
described 

—  The  conditions  that  caused  an  issue  to  occur  along  with  root  cause(s) 
need  to  be  identified  and  described 
•  This  includes  the  consequences  of  the  issue  occurrence 

-  The  risk  writing  guidelines  presented  earlier  can  be  used  as 
appropriate  to  issues 
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Issue  Analysis 

•  Issue  analysis  activities  are  similar  to  those  for  risk  analysis 

-  If  an  issue  is  a  risk  that  has  occurred,  some,  if  not  all,  risk  analysis  may 
be  sufficient  for  the  issue 

•  The  issue  may  need  additional  analysis  due  to  its  impact  on 
program/project  activities  and  products 

-  If  the  issue  is  one  that  occurred  without  it  being  associated  with  a  risk 
then  issue  analysis  needs  to  be  conducted 

•  The  root  cause  needs  to  be  identified 

•  Impact  level,  rated  from  very  low  (1)  to  very  high  (5),  is  assessed  in  at 
least  four  categories: 

-  Cost 

-  Schedule 

-  Scope 

-  Performance 
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Issue  Response 

•  Issue  response  could  include 

—  Acceptance 

•  Accept  the  consequence  of  the  issue 

•  Same  reasons  as  acceptance  of  risks 

-  Transfer 

•  If  the  occurred  risk  had  not  been  transferred  the  issue  responsibility  may 
now  be  shifted  to  another  party  who  is  better  equipped  to  deal  with  the 
issue 

-  Escalate 

•  If  the  occurred  risk  had  not  been  escalated  the  issue  responsibility  may 
now  be  escalated  to  higher  authority  based  on  threshold  described  earlier 

-  Contingencies 

•  Can  be  developed  using  the  similar  methods  described  for  risk  mitigations 
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Issue  Response 

•  Issue  contingencies  are  strategies  to  eliminate  or  reduce  the 
negative  effect  of  the  issue 

•  Issue  contingencies  should  have  been  developed  for  issues 
that  were  the  result  of  a  risk  occurring 

—  If  not  they  need  to  be  developed 
—  They  may  need  to  be  enhanced 

•  If  they  were  associated  with  risks  earlier 

•  For  issues  that  were  not  the  result  of  a  risk  occurring 
contingencies  need  to  be  developed 
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Issue  Monitoring  and  Control 

•  Issues  are  monitored  and  controlled  until  they  are  closed 

•  Issues  are  monitored  and  controlled  using  similar  methods 
described  for  risk  monitoring  and  control 
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Ice  Cream  Risk/Issue 


Probability  of  occurrence  97%  -  Very  High 
Consequence  of  occurrence  -  Very  High 
isk  Priority  -  Very  High 


Mommy, 

mommy  i  want  tingency  —  Pray 

ice  cream 


Signal  Broken 


Car  stalled  on  tracks 
Battery  still  dead 
Doors  jammed 
Widows  Jamm 


Train  approaching  200  yards  away  going  30  mph 


Murphy  is  ^ 
for  real!  j 
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Black  Swan  Events 


The  Black  Swan:  The  Impact  of  the  Highly  Improbable ; 

Nassim  Nicholas  Taleb 

WikipediA 
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Black  Swan  Events 


•  Black  Swan  Events  were  described  by  Nassim  Nicholas  Taleb 
in  his  2007  book,  The  Black  Swan 

•  Taleb  regards  almost  all  major  scientific  discoveries, 
historical  events,  and  artistic  accomplishments  as  "black 
swans"— undirected  and  unpredicted 

•  He  gives  the  rise  of  the  Internet,  the  personal  computer, 
World  War  I,  and  the  September  11,  2001  attacks  as 
examples  of  Black  Swan  Events 


We  can  add  the  2004  Indonesian  Tsunami ',  The  Haiti  Earthquake ,  Columbia , 
Challenger,  Killer  Whale  Tilikum,  Toyota  Recalls-to  Stock  Holders  and 
Owners,  not  sure  about  Toyota  Executives  -  Al  Florence 


y&/  Florence 


91 


Black  Swan  Theory 

•  The  Black  Swan  Theory  is  used  by  Nassim  Nicholas  Taleb  to 
explain  the  existence  and  occurrence  of  high-impact,  hard- 
to-predict,  and  rare  events  that  are  beyond  the  realm  of 
normal  expectations 

•  Such  events  are  considered  extreme  outliers 
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Black  Swan  Events 


•  The  main  idea  in  Taleb's  book  is  not  to  attempt  to  predict  Black  Swan 
Events 

-  but  to  build  robustness  into  negative  ones  that  occur  and  being  able  to 
exploit  positive  ones 

•  Taleb  contends  that  banks  and  trading  firms  are  very  vulnerable  to 
hazardous  Black  Swan  Events  and  are  exposed  to  losses  beyond  that 
predicted  by  their  defective  models 

—  Sounds  familiar?  Al  Florence 

•  Taleb  states  that  a  Black  Swan  Event  depends  on  the  observer 

-  using  a  simple  example,  what  may  be  a  Black  Swan  surprise  for  a  turkey  is 
not  a  Black  Swan  surprise  for  its  butcher 

-  hence  the  objective  should  be  to  "avoid  being  the  turkey"  by  identifying 
areas  of  vulnerability  in  order  to  "turn  the  Black  Swans  white" 


The  same  can  be  said  about  9/11;  Black  Swan  for  Americans, 
White  Swan  for  terrorists!  -  Al  Florence 
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Black  Swan  Bottom  Line 


•  Black  Swan  Events  are  events  that  have  an  extremely  low 
probability  of  occurrence 

—  cannot  be  predicted 

•  But  have  a  very  high  consequence  if  occur 

—  positive  or  negative 

•  Mitigation  is  near  impossible  for  negative  ones  since  it  is  not 
known  when,  where  and  how  or  if  they  will  occur 

Al  Florence 


How  about  an  All  Black  Penguin? 
Discovered  March  11,  2010  near  Antarctica 
One  in  a  zillion  occurrence 
So,  what  would  a  All  Black  Penguin  event  be? 
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Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

•  Reasons  for  Risk/Issue  Management 

•  Opportunities 

•  Risk  Management 

•  Issue  Management 
^  •  Risk/Issue  Avoidance 

•  Risk/Issue  Opportunities 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Risk/Issue  Avoidance 


•  Risk/Issue  Avoidance  has  at  least  two  components: 

-  Eliminate  the  sources  of  high  risks  and  issues  and  replace  them 
with  a  lower-risk  alternatives 

-  The  establishment  of  sound  technical,  programmatic  and 
management  processes,  and  activities  early  and  their  continued 
execution  throughout  the  entire  lifecycle 

•  This  helps  remove  common  risks/issues  root  causes 
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Low-Risk  Alternatives 


•  Eliminate  the  sources  of  high  risk  and  replace  them  with  a 
lower-risk  alternatives 

-  The  selection  and  implementation  of  alternative  activities  or  products 
without,  or  with  lower,  risk  to  replace  the  riskier  activities  or  products 

—  These  alternatives  may  eliminate  or  reduce  the  impact  if  the  risk 
occurs 
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Alternative  Approaches 


•  Risk  analysis  and  mitigation  involves  the  investigation  and 
implementation  of  alternative  approaches  which  may  include: 

-  Technical 

•  Example:  substitute  COTS  products,  relax  requirements 

-  Management 

•  Example:  reassign  tasks,  augment  teams 

-  Programmatic 

•  Example:  manipulate  schedules,  reduce  documentation, 
increase  budgets 
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Industry  Best  Practices 


•  Implementing  industry  best  practices  within  organizations  and 
executing  them  on  projects  can  drastically  reduce  risks  and 
issues 

•  Best  practices  include  technical  and  programmatic  processes 
and  procedures 


By  doing  this  organizations  will  go  a  long  way  in 

reducing  risk  and  issues 
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SEI  CMMI 


•  Software  Engineering  Institute  (SEI)  Capability  Maturity 
Model  Integration  (CMMI) 

—  CMMI  for  Development  vl.3-  For  suppliers  developing/supplying 

•  Software 

•  Hardware 

•  Systems 

•  Commercial  of  the  Shelf  (COTS) 

—  CMMI  for  Acquisition  vl.3 

•  For  customers  acquiring  systems,  products  and  services 

—  CMMI  for  Service  vl.3 

•  For  organizations  providing  services 

All  these  CMMIs  have  process  areas  for  most  activities  associated  with 
program/project  development  and  acquisition 

Their  proper  implementation  would  result  in  best  practices 
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SEI  CMMI 


CMMI  Process  Areas  for  CMMI  for  Development  vl.2 


Level 

Process  Areas 

5  -  Optimizing 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

4-  Quantitative  Managed 

Organizational  Process  Performance 

Quantitative  Project  Management 

3  -  Defined 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

2  -  Managed 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

1  -  Initial  Competent  people  and  heroics,  sometimes  ad  hoc  (Subject  to  high  incidence  of  RISKS) 

Florence 


SEI  CMMI 


The  CMMI  is  itself  a  Risk  Management  Plan 

Managing  Risks ,  Methods  for  Software  Systems  Development;  Dr.  Elaine  M.  Hall, 

SEI  Series  in  Software  Engineering 
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Other  Frameworks 


•  Other  Methodologies/Frameworks 

—  6  Sigma 

•  Process  Improvement 

•  Reduce  variation 

•  Statistical/quantitative 

-  Lean  6  Sigma 

•  Process  Improvement 

•  Reduce  Waste 

•  Less  statistically  rigorous  then  6  sigma 

—  ITIL-  Information  Technology  Infrastructure  Library 

•  For  Information  Technology 

•  Services  Focused 
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Other  Frameworks 

ISO  9000  standards 

•  ISO  9001  -  applies  to  companies  required  to  design,  develop, 
produce,  install,  service  and  supply  a  product  or  service 

•  ISO  9002  -  applies  to  companies  required  to  produce,  install, 
service  and  supply  a  product  or  service  according  to  an 
existing  design 

•  ISO  9003  -  addresses  only  the  requirements  for  detection  and 
control  of  problems  during  final  inspection  and  testing 

Others 
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Reducing  Risks  with  the  Proper 
Specification  of  Requirements 


The  next  presentation  was  published  as  a  paper  in  Crosstalk,  The  Journal  of 
Defense  Software  Engineering,  April  2002  by  Al  Florence 

Title  of  the  publication: 

RISKY  Requirements 

It  also  received  an  award  in  the  best  paper  competition  at  MITRE 


December  1 7,  2008  —  Computerworld  — 

Melinda-Carol  Ballou,  an  analyst  with  IDC  in  Framingham,  Mass.,  said 
that  70%  to  80%  of  IT  project  failures  relate  directly  to  poor  requirements 
gathering,  management  and  analysis. 

And  specification  -  Al  Florence 
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Proper  Specification  of  Requirements 

•  A  government  agency,  while  modernizing  their  information  systems, 
reverse-engineered  requirements 

•  With  domain  knowledge  of  the  application,  several  teams  were 
involved 

-  They  represented 

•  Users 

•  Contractors 

•  Acquisition  organization 

•  This  author  was  assigned  as  a  consultant  to  guide  the  teams  in  the 
proper  specification  of  requirements 

•  The  examples  presented  show  some  of  the  requirements: 

-  As  initially  specified  by  the  teams 

-  Next  a  critique  of  the  requirements  by  this  author 

-  Finally  the  re-specified  requirements  based  on  the  critique 
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f  Criteria  for  Specifying  a  Good 

Requirement 

The  following  are  some  critical  attributes  that  requirements  must 
adhere  to:  (Used  to  critique  requirements) 

•  Completeness:  Requirements  should  be  complete 

They  should  reflect  system  objectives  and  specify  the  relationship 
between  the  software  and  the  rest  of  the  subsystems 

•  Consistency:  Requirements  must  be  consistent  with  each 
other;  no  requirement  should  conflict  with  any  other 
requirement 

Requirements  should  be  checked  by  examining  all  requirements  in 
relation  to  each  other  for  consistency  and  compatibility 
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Criteria  for  Specifying  a  Good 

Requirement 

•  Traceability:  Each  requirement  must  be  traceable  to  some 
higher-level  source,  such  as  a  system  level  requirement 

Each  requirement  should  also  be  traced  to  lower  level  design  and  test 
abstractions  such  as  high-level  and  detailed-level  design  and  test  cases 

•  Testability:  All  requirements  must  be  testable  in  order  to 
demonstrate  that  the  software  end  product  satisfies  its 
requirements 

In  order  for  requirements  to  be  testable  they  must  be  specific , 
unambiguous ,  and  quantitative  whenever  possible.  Avoid  negative , 
vague  and  general  statements 


Z\£<jTr&^ &/  Florence 


108 


Criteria  for  Specifying  a  Good 

Requirement 

•  Feasibility:  Each  requirement  must  be  feasible  to  implement 

Requirements  that  have  questionable  feasibility  should  be  analyzed 
during  requirements  analysis  to  prove  their  feasibility 

•  Unique  identification:  Uniquely  identifying  each  requirement 
is  essential  if  requirements  are  to  be  traceable  and  testable 

Uniqueness  also  helps  in  stating  requirements  in  a  clear  and  consistent 
fashion 


&/  Florence 


109 


Criteria  for  Specifying  a  Good 

Requirement 

•  Design  Free:  Software  requirements  should  be  specified  at  a 
requirements  level  not  at  a  design  level 

The  approach  should  be  to  describe  the  software  requirement  functionally  from  a 
system  (external)  point  of  view,  not  from  a  software  design  point-of-view,  i.e. 
describe  the  system  functions  that  the  software  must  satisfy.  Some  requirements 
may  have  design  embedded  due  to  constraints  placed  on  them  by  the  system, 
interfaces  or  legacy 

•  Use  of  "shall"  and  related  words:  In  specifications,  the  use  of  the  word 
"shall"  indicates  a  binding  provision 

Binding  provisions  must  be  implemented  by  users  of  specifications.  To  state  non¬ 
binding  provisions,  use  "should"  or  "may".  Use  "will"  to  express  a  declaration  of 
purpose  (e.g.,  "The  Government  will  furnish... "),  or  to  express  future  tense.  MIL- 
STDs 

Note:  Methods  other  that  the  use  of  ‘‘shall”  can  be  used  to  specify  requirements  such  as 
using  a  matrix  with  a  column  for  requirements  and  another  column  for  comments  or  italics 
or  underlines  for  comments. 
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Example  1 


•  Initial  specification 

The  Financial  Agent  sends  to  the  government  by  6:00  PM  ET  on  the 
same  day  after  receipt  of  the  file  CRDF  that  includes  only  critical  data 
collected  from  the  enrolled  individual. 

•  Critique 

-  No  unique  identifier  provided 

-  The  word  "shall"  is  missing 

-  How  is  the  file  sent? 

-  Has  design  implications:  "CRDF" 

•  Should  define  data,  not  name  of  data  file  -  this  should  be  done  in 
the  design 

-  The  critical  data  has  to  be  identified 
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Example  1 


•  Re-specification 

3.3. 1.3  The  Financial  Agent  shall  send  the  government,  via 
electronic  transmission,  the  following  critical  data  collected 
from  each  enrolled  individuals  by  6:00  PM  ET  on  the  day  of 
receipt: 

a.  Name 

b.  Address 

c.  Zip  Code 

d.  Social  Security  Number 
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Example  2 

•  Initial  specification: 

3. 2. 5. 7  The  system  shall  process  two  new  fields  (provides 
production  count  balancing  info  to  states)  at  the  end-of-state 
record 

•  Critique: 

-  This  requirement  cannot  be  implemented  or  tested. 

-  It  is  incomplete.  What  are  the  two  new  fields? 

-  "Info"  should  be  spelled  out 

•  Re-specification: 

3. 2. 5. 7  The  system  shall  provide  the  following  data  items 
(provides  production  count  balancing  information  to  states)  at 
the  end-of-state  record: 

a.  SDATE,  and 

b.  YR-TO-DATE-COUNT 
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Example  2 


•  Re-Critique: 

-  This  rewrite  has  design  implications  SDATE  record  and  YR-TO-DATE- 
COUNT 

•  From  a  requirements  viewpoint  it  should  specify  what  the  data  in 
the  records  are,  not  the  name  of  the  record  as  it  exists  in  the 
design  and  implementation 

•  Re-Re-Specification: 

3. 2. 5. 7  The  system  shall  provide  the  following  data  items  (provides 
production  count  balancing  information  to  states)  at  the  end-of-state 
record: 


a.  Submission  date  and  time 

b.  Year-to-date  totals 
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Example  3 

Initial  specification: 

3. 2. 5. 9  All  computer-resident  information  that  is  sensitive  shall  have 

system  access  controls.  Access  controls  shall  be  consistent  with  the 
information  being  protected  and  the  computer  system  hosting  the  data. 

Critique: 

-  Two  "shalls"  under  one  identifier 

-  The  requirement  is  vague  and  incomplete.  Need  to  identify  the  sensitive 
information. 

-  What  does  "consistent"  mean? 

-  As  specified  it  cannot  be  implemented  or  tested 

Re-specification: 

3. 2. 5. 9  All  sensitive  computer-resident  information  shall  have  system 
access  controls,  consistent  with  the  level  of  protection.  (Reference 
Sensitive  Information,  Table  5.4.1  and  Level  of  Protection  for  Sensitive 
Information,  Table  5.4.2) 
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Example  4 


•  Initial  specification: 

Software  will  not  be  loaded  from  unknown  sources  onto  the  system 
without  first  having  the  software  tested  and  approved 

•  Critique: 

-  If  it  is  tested  and  approved,  can  it  be  loaded  from  unknown  sources? 

-  If  the  source  is  known,  can  it  be  loaded  without  being  tested  and 
approved? 

-  Requirement  is  ambiguous  and  stated  as  a  negative  requirement, 
which  makes  it  difficult  to  implement  and  test 

-  A  unique  identifier  is  not  provided,  which  makes  it  difficult  to  trace 

-  The  word  "shall"  is  missing 
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Example  4 


•  Re-specification: 

3. 2. 5. 2  Software  shall  be  loaded  onto  the  operational  system  only 
after  it  has  been  tested  and  approved 
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Example  5 


•  Initial  specification: 

3.2.7. 1  The  system  shall  purge  state  control  records  and  files  that  are  older 
than  the  operator  or  technical  user-specified  retention  period 

•  Critique: 

-  Requirement  is  incomplete  and  vague  without  specifying  the 
retention  period  or  providing  a  reference  as  to  where  the  information 
can  be  obtained 

-  Requirement  cannot  be  implemented  or  tested  as  stated 

•  Re-specification: 

3. 2. 7.1  The  system  shall  purge  state  control  records  and  files  that  are  older 
than  the  retention  period  input  into  the  system  by  either  the 

a.  operator,  or 

b.  technical  user 
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Example  6 


•  Initial  specification: 

3. 3. 2.1  The  system  shall  have  no  single  point  failures 

•  Critique: 

-  This  is  an  ambiguous  requirement.  Needs  identification  of  what 
components  and/or  functions  the  "no  single  point  failures"  applies  to 

-  As  specified  it  cannot  be  implemented  or  tested 

•  Re-specification: 

3. 3. 2.1  The  following  system  components  shall  have  no  single  point 
failures.  a>  Host  servers  e.  Hubs 


b.  Networks 

c.  Network  routers 

d.  Access  servers 


f.  Switches 

g.  Firewalls 

h.  Storage  devices 
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Example  7 


•  Initial  specification: 

3. 2. 6. 3  The  system  shall  receive  and  process  state  return  data  from  the 

State  Processing  Subsystem.  The  system  shall  provide  maintenance  of  the 
state  data  files  and  generate  various  reports. 

•  Critique: 

-  Two  "shall s"  under  one  requirement  number  and  multiple 
requirements  in  the  specification 

-  The  word  "process"  in  the  first  shall  is  vague;  need  to  define  the 
processing  required 

-  The  second  "shall"  does  not  provide  for  valid  requirements;  they 
cannot  be  implemented  or  tested  as  stated 

•  Needs  identification  of  type/amount  of  maintenance  required 

•  "various  reports"  is  ambiguous 
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Example  7 


Re-specification: 

3.2.63  The  system  shall  receive: 

a.  production  data  that  contains  data  from  multiple  states,  and 

b.  state  total  amount  for  one  or  more  states, 
extracted  by  the  Returns  Processing  Subsystem. 

3. 2. 6.4  The  system  shall  parse  multi-state  data  to  respective  state  files 

3. 2. 6. 5  The  system  shall  display  a  summary  screen  reporting  the  results 
of  processing  for  each  state  containing: 

a.  state  totals, 

b.  state  generic  totals,  and 

c.  state  unformatted  totals. 
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Example  8 


•  Initial  specification: 

3.2.9. 1  When  doing  calculations  the  software  shall  produce  correct 
results. 

•  Critique: 

-  Really?  This  is  not  a  requirement. 

-  This  type  of  non-requirements  should  not  be  specified! 

-  It  should  be  deleted. 

•  Re-specification: 

Requirement  deleted. 
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Where  Are  We 


•  Tutorial  Objectives 

•  Introduction 

•  Reasons  for  Risk/Issue  Management 

•  Opportunities 

•  Risk  Management 

•  Issue  Management 

•  Risk/Issue  Avoidance 

■=^  •  Risk/Issue  Opportunities 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Risk/Issue  Opportunity 

•  An  Opportunity  is  a  potential  event  which,  if  taken  advantage 
of,  will  positively  affects  the  ability  of  the  project  or  program  to 
meet  its  objectives 

-  Opportunities  to  improve  project  performance  may  surface  while 
applying  mitigation  or  contingency  strategies 

-  The  mitigation  or  contingency  strategies  could  be  used  as  best 
practices  on  the  project  or  other  projects  to  keep  the  risk  from 
recurring 
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Risk  /  Opportunity 


•  An  opportunity  may  be  the  potential  of  improving  the  value  of 
project  results;  the  project  itself  is  the  paramount  opportunity 

•  A  risk  is  the  potential  of  not  achieving  the  project  strategic 
opportunity 

•  Taking  risks  can  lead  to  opportunities  but  risk  taking  can  itself 
be  quite  risky  unless  the  risk  is  well  thought  out  and  well 
managed 
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Capturing  Lesson  Learned  in  Support  of 

Opportunities 


Contingency  Strategy  Reduced  Impact 


Stored  in  lessons  learned  database  which  can  be  documented  as  a  best  practice 
opportunity  to  be  reused  on  this  or  other  projects  to  keep  the  risk  or  issue  from  recurring 
(Hopefully  Identified  and  corrected  Root  Cause)  (  Also  Called  Process  Improvement) 


Z\£<jTT&^ &/  Florence 


126 


Real  Project  Example 

•  Opportunity 

-  Galileo  Space  Probe  to  Jupiter  to  investigate  the  Jovian  atmosphere  and 
gather  scientific  information 

•  A  Risk 

-  Voyager  Space  Craft  on  flyby  of  Jupiter  discovered  high  radiation,  which 
destroys  electrical  components,  and  high  incidences  of  cosmic  ray  events, 
which  flips  volatile  computer  bits 

-  IF  the  Galileo  Probe  is  not  designed  to  protected  against  this  phenomena 
prior  to  launch 

-  THEN  the  microprocessor  may  fail  before  the  mission  is  accomplished 
resulting  in  less  science  collected  or  complete  mission  failure 

•  A  mitigation  strategy  was  developed  and  deployed 

-  Redesign  the  configuration  of  the  probe  to  protect  volatile  bits  and 
electronic  components 

•  Issue 

-  Did  not  materialize 

•  Opportunities 

-  Galileo  Probe  was  successful  and  accomplished  its  full  mission 

-  Lessons  learned  were  used  on  other  space  missions 
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Real  Project  Example 


•  Mitigation 

-  Initial  design  had  one  microprocessor  to  command  and  control  the 
instruments,  collect  scientific  data  and  telemeter  to  orbiter 

-  Eliminate  2  experiments  and  install  2nd  microprocessor 

•  Both  redundantly  collecting  same  data  and  sending  to  orbiter 

•  Place  processors  in  the  center  of  probe 

-  Provide  for  redundant  triple  collection  and  triple  voting  on  both 
processors  on  critical  parameters  such  as  timing 

-  Hopefully  when  one  processor  destroyed  the  other  may  continue  to 
operate 


This  mitigation  (redesign)  extended  the  life  of  science  gathering  until  probe  implosion 


Z\£<jTT&^ &/  Florence 


128 


Ice  Cream  Issue/Opportunity 


Too  late  for 
Contingencies 


Probability  of  occurrence  100%;  risk  occurred 

Now  this  is  a  real  issue! 


Harvest  lessons  learned  for  Improvement 
Opportunities 

(Fix  signal  and  install  closing  gates  on  both  sides  of 
tracks,  examine  breaks  and  doors  on  cars) 


The  good  news  is  that  no  one  was  hurt, 
obviously  this  is  a  toy  train  and  toy  cars 
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Where  Are  We 


Tutorial  Objectives 
Introduction 

Reasons  for  Risk/Issue  Management 

Opportunities 

Risk  Management 

Issue  Management 

Risk/Issue  Avoidance 

Risk/Issue  Opportunities 

Questions/Discussion 

References 

Contact  Information 
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Software  Safety 

What  Does  it  Mean? 
How  Do  We  Do  It? 
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Software  and  Systems  Safety 


Creating  a  Safe  System  is  a  “team  effort” 

Software  is  a  vital  part  of  most  systems 

-  What  hardware  and  software  does  it  control? 

-  How  do  we  know  if  our  software  is  “safe”  or  “unsafe”? 

-  What  are  the  hazards  that  software  contributes  to  or  that 
software  controls? 
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Why  Should  Anyone  Care 
About  Software  Safety? 


Why  should  anyone  care  about  software  safety? 

-  When  a  device  or  system  can  lead  to  injury,  death,  the 
destruction  of  loss  of  vital  equipment  or  damage  to  the 
environment,  software  safety  is  critical! 

-  Software  can  respond  quickly  to  potential  problems, 
provide  more  functionality  than  equivalent  hardware  and 
can  even  be  changed  in  real  time 

•  Software  is  flexible  as  long  as  those  who  develop  it  and 
integrate  it  into  the  other  system  components  understand  the 
consequences  of  its  possible  failure 

-  Software  safety  includes  all  software  that  can  impact 
hazardous  software  or  hardware 

•  All  such  software  is  “safety-critical” 
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Hazardous  and  Safety-Critical  Software 
Definitions 


Hazard  -  A  hazard  is  the  presence  of  a  potential  risk 
situation  that  can  result  in  or  contribute  to  a  mishap 

-  Every  hazard  has  at  least  one  cause  which  can  turn  into  a 
number  of  effects  (damage,  illness,  failure) 

Hazard  Cause  -  A  hazard  cause  may  be  a  defect  in 
hardware  or  software,  a  human  operator  error  or  an 
unexpected  input  or  event  which  results  in  a  hazard 

Hazard  Control  -  A  hazard  control  is  a  method  of 
preventing  the  hazard,  reducing  the  likelihood  of  the  hazard 
occurring,  or  the  reduction  of  the  impact  of  that  hazard 
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Hazardous  and  Safety-Critical  Software 
Definitions  -  2 


Hazard  Controls  use: 

-  Hardware  (e.g.,  pressure  relief  or  valve) 

-  Software  (e.g.,  detection  of  stuck  valve  and  automatic 
response  to  open  secondary  valve 

-  Operator  procedures 

-  Combination  of  methods  to  avert  the  hazard 

Hazard  Control  Method-  For  every  hazard  cause,  there 
must  be  at  least  one  control  method  which  is  usually  a 
design  feature  or  a  procedural  step 
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Hazardous  and  Safety-Critical  Software 
Definitions  -  3 


Hazard  Control  Verification  -  Each  hazard  control  will 
require  verification 

-  Test 

-  Analysis 

-  Inspection 

-  Demonstration 

Software  -  Software  can  be  used  to  detect  and  control 
hazards,  but  software  failures  can  also  contribute  to  the 
occurrence  of  hazards! 
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Safety-Critical  Software 


Safety-critical  software  includes: 

-  Hazardous  software  which  can  directly  contribute  to, 
or  control  a  hazard 

-  All  software  that  influence  this  hazardous  software 

-  Software  that  monitors  hazardous  or  safety-critical 
hardware  or  software 

EXAMPLE:  Software  that  controls  a  high-powered 
laser  is  hazardous  and  safety-critical 
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Safety-Critical  Software  -  2 


Software  that  provides  information  required  for  a 
safety-related  decision  falls  into  the  safety-critical 
category 

-  If  a  human  must  shut  down  a  piece  of  hardware  when  the 
temperature  goes  over  a  threshold,  the  software  that  reads 
the  temperature  and  displays  it  for  the  human  operator  is 
safety-critical 

-  All  the  software  along  the  chain,  from  reading  the 
hardware  temperature  sensor,  converting  the  value  to 
appropriate  units,  to  displaying  the  data  on  the  screen  are 
safety-critical 
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Safety-Critical  Software  -  3 


Software  is  safety-critical  if  it  performs  any  of  the 
following: 

-  Controls  hazardous  or  safety-critical  hardware  or 
software 

-  Monitors  safety-critical  hardware  or  software  as  part 
of  a  hazard  control 

-  Provides  information  upon  which  a  safety-related 
decision  is  made 

-  Verifies  hardware  or  software  hazard  controls 

-  Can  prevent  safety-critical  hardware  or  software  from 
functioning  properly 
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Software  Safety  Program 


A  System  Safety  Program  Plan  is  a  prerequisite  to 
performing  development  or  analysis  of  safety-critical 
software.  The  Safety  Plan  should: 

-  Describe  forms  of  analysis 

-  Provide  a  schedule  for  performing  a  series  of  these  system 
and  subsystem  level  analysis  throughout  the  development 
schedule 

System  safety  analysis  follow  the  lifecycle  of  the  system 
development  efforts 

-  The  system  is  compromised  of  the  hardware,  the  software, 
and  the  interfaces  between  them,  including  human 
operators 
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System/  Software 
Safety  Lifecycles 
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Systems  Lifecycle 
Software  Cycle 
Software  Safety 
Lifecycle  Overlay 
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Systems  /  Software 
Product  Lifecycles 
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Preliminary  Hazard  Analysis 
Systems  Require  Analysis 


Architectural  Design 


Software  Safety 

_ / 

SW  Subsystem  Hazard  Analysis 


Software  Require  Analysis 


Detailed  Design 


Software  Failure  Modes  and 
Effects  Analysis 


Coding  &  Unit  Testing 


Implement  Safety  Controls 


Integration  & 
Systems  Testing 


Safety  Testing  of  Interfaces 
Testing  of  Software  that  is  Safety-Critical 
Testing  of  Non-Safety-Critical  Code 
That  Can  Compromise  Safety  Controls 
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Safety  Requirements 
and  Analysis 


The  first  step  to  any  safety  analyses  is  asking  the  right 
questions: 

-  What  could  go  wrong? 

-  Why  won’t  it? 

-  What  if  it  did? 

Everyone  involved  in  each  activity  throughout  the  product 
lifecycle  should  think  about  all  the  ways  the  system  or 
software  can  fail,  how  to  prevent  the  failure  from  occurring 
and  how  to  prevent  or  mitigate  an  accident  if  the  failure 
occurred 
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Safety  Requirements 
and  Analysis  -  2 


In  general  there  are  two  types  of  safety  requirements: 

-  Imposed  by  standards  and  regulations 

-  Technically  specific  to  the  project  and  its  operating 
environments 

The  most  commonly  used  assessment  tool  used  during  this 
beginning  activity  is  the  Preliminary  Hazard  Analysis  (PHA) 
as  illustrated  in  the  Software  Safety  Lifecycle  Overlay 
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Preliminary  Hazard 
Analysis  (PHA) 


Before  any  system  with  software  can  be  analyzed  or 
developed  for  use  in  hazardous  operations  or 
environments,  a  system  PHA  must  be  performed 

The  PHA  is  the  first  source  of  “specific”  system  safety 
requirements  which  may  go  to  the  level  of  pointing  out  the 
specific  software  safety  requirements 

When  a  system  is  decomposed  into  subsystems,  how  the 
software  relates  to  each  component  must  be  considered 
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Software  Subsystem 

Hazard  Analysis 


Software  Subsystem  Hazard  Analysis  determines  which  of 
the  hazards  including  hazard  causes,  hazard  controls, 
hazard  mitigations,  or  hazard  verifications  have  software  as 
a  component 

Software  may  impact  a  hazard  in  a  variety  of  ways: 

-  Software  failure  may  lead  to  a  hazard  occurring  (Hazard 
Cause) 

•  A  failure  in  a  data  conversation  function  could  incorrectly 
report  a  temperature  valve  resulting  in  personal  damage 

-  Failure  of  a  Software  Hazard  May  Allow  a  Hazard  to  Occur 

•  Failure  that  monitors  pressure  and  opens  a  valve  when  it 
reaches  a  threshold  may  be  a  hazard  control 

•  Failure  of  that  software  would  allow  an  over-pressurization  of 
the  vessel  and  a  potential  for  a  rupture  or  other  hazard 
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Software  Subsystem 

Hazard  Analysis  -  2 _ 

-  Software  Safing  Mode  May  Fail 

•  Failure  of  the  software  to  detect  or  shut  down  a 
runaway  electromechanical  device  is  an  example  of 
such  an  impact 

-  Software  Used  to  Mitigate  the  Consequences  of  An 
Accident  May  Fail 

•  Software  controlling  the  purging  of  toxic  gas  may  fail, 
allowing  the  gases  to  remain  in  the  chamber  or  be 
vented  inappropriately  to  the  outside  air 

-  Software  Used  to  Verify  Hazard  Hardware  or 
Software  Hazard  Controls  May  Fail 

•  Failure  in  this  situation  would  be  due  to  invalid  results 

•  False  positives  may  allow  a  potentially  hazardous 
system  to  be  put  into  operation 
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Software  Safety  Planning 


If  the  Preliminary  Hazard  Analysis  (PHA)  reveals  that  any 
software  is  safety-critical,  a  software  safety  process  must 
be  initiated 

Standard  Participant’s  in  a  Successful  Safety  Process 
include: 

-  Project  Manager 

-  Systems  Engineers 

-  System  Safety  Engineers 

-  Software  Safety  Engineers 

-  Software  Developers 

-  Software  Quality  Assurance 

-  Independent  Verification  &  Validation 
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Standard  Participants  in  a 

Successful  Safety  Process _ 

Project  Manager 

-  The  Project  Manager  maintains  a  high-level  overview 
of  the  whole  process,  works  with  the  team  to  include 
the  extra  software  development  and  analysis  tasks  in 
the  schedule  and  budget,  and  can  be  called  upon  to 
help  in  negotiations  between  team  members 

-  Ultimately,  the  Project  Manager  carries  the 
responsibility  for  determining  the  amount  and  types  of 
risk  they  are  willing  to  accept  for  their  project 
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Standard  Participants  in  a 
Successful  Safety  Process  -  2 


Systems  Engineers 

-  Systems  Engineers  are  responsible  for  designing  the 
system  and  partitioning  the  elements  between  hardware 
and  software. 

-  Systems  Engineers  own  the  “big  picture” 

-  Systems  developers  can  make  sure  that  safety-critical 
elements  are  not  overlooked  in  the  initial  analysis 

-  Systems  Engineers  makes  sure  that  “criticality”  of  the 
design  component  is  transferred  to  software  later  in  the 
design  process 
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Standard  Participants  in  a 

Successful  Safety  Process  -  3 _ 

System  Safety  Engineers 

-  System  Safety  Engineers  determine  what  components  of 
the  systems  are  hazardous 

-  They  look  at  the  system  as  a  whole  entity,  but  also  try  to 
understand  how  the  different  elements  function  or  fail  to 
function 

Software  Safety  Engineers 

-  Software  Safety  Engineers  build  on  the  work  of  the  system 
safety  engineers 

-  They  look  closely  at  the  requirements  and  development 
process  to  verify  that  software  safety  requirements  are 
present  and  being  designed  in 

-  They  perform  analyses,  and  tests  to  verify  the  safety  of  the 
software 
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Standard  Participants  in  a 

Successful  Safety  Process  -  4 _ 

Software  Developers 

-  Software  Developers  are  the  “in  the  trenches”  engineers 
who  design,  code,  and  often  test  the  system 

-  Software  developers  can  have  a  great  impact  on  the  safety 
of  the  software  by  the  design  chosen  and  by  implementing 
defensive  programming  techniques 

Software  Quality  Assurance 

-  Software  Quality  Assurance  engineers  work  with  the 
software  developers  and  software  safety  engineers  to 
assure  that  safe  software  is  created 

-  Software  Quality  Assurance  monitors  development 
processes,  assures  the  traceability  of  all  requirements, 
performs  or  reviews  analyses,  witnesses  or  performs  tests, 
and  provides  an  “outside”  view  of  both  the  software 
process  and  products 
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Standard  Participants  in  a 
Successful  Safety  Process  -  5 _ 

Independent  Verification  &  Validation 

-  For  all  safety  critical  projects,  the  Project  Manager  is 
required  to  perform  a  self  assessment  of  the  project 
software  and  report  the  findings 

-  IV&V  may  then  perform  their  own  review  and  present 
the  Project  Manager  with  their  estimate  for  additional 
analyses 

-  IV&V  is,  in  addition  to  Software  Quality  Assurance 
and  Software  Safety  and  not  a  replacement  for  those 
roles 
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Identifying  Safety 
Critical  Software 
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Identifying  Safety-Critical  Software 


Software  that  can  cause  a  hazard  if  it  malfunctions  is 
considered  safety-critical  software 

-  Software  which  is  required  to  recognize  hazardous 
conditions 

-  Software  which  implements  automatic  safety  control 
actions 

-  Software  which  provides  a  safety-critical  service 

-  Software  which  inhibits  a  hazardous  event 
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Complexity 


Greater  complexity  of  the  software  system  increases 
the  chances  of  errors 

-  Software  complexity  is  increased  as  the  number  of 
safety  related  software  requirements  for  hazards 
control  increases 

-  Rough  measures  of  complexity  include  the  number  of 
subsystems  controlled  by  the  software  and  the 
number  of  interfaces  between: 

•  Software  to  Software 

•  Software  to  User 

•  Software  to  Software  Subsystems 
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Time  Criticality 


The  speed  at  which  a  system  with  software  hazard 
control  must  react  is  a  major  factor 

-  Does  control  need  to  be  exercised  faster  than  a 
human  or  mechanical  system  can  react? 
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System  and  Software 
Concept  Phase 
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Initial  Planning  of  the  Software 
Development  Effort 


During  the  System  Concept  phase,  the  software  team 
should  involved  in  the  initial  planning  of  the  software 
development  effort 

Plans  typically  developed  in  the  Concept  Phase  include: 

-  Software  Project  Plan  /  Software  Management  Plan 

-  Software  Development  Plan 

-  Software  Quality  Assurance  Plan 

-  Software  Configuration  Management  Plan 

-  Software  Safety  Plan 

-  Software  Verification  and  Validation  Plan 

-  Software  Acquisition  Plan 
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System  Concept  Phase  Tasks 


Software  Engineering  Tasks 

System  and  Software  Safety  Tasks 

Provide  input  to  project  concept  and  software 
concept  documents 

Create  the  Software  Safety  Plan,  including  planning 
and  tailoring  the  safety  effort. 

Provide  input  to  Software  Safety  Plan 

Conduct  Preliminary  Hazard  Analysis  (PHA) 

Plan  the  software  management  and  development 

processes 

Set  up  hazards  tracking  and  problem  resolution 

process 

Plan  the  Configuration  Management  System 

Prepare  hazard  verification  matrix 

Plan  the  verification  and  validation  process 

Review  PHA  for  safety-critical  software 

Participate  in  "make  or  buy"  decisions  for  software 
Review  software  acquisition 

Participate  in  "make  or  buy"  decisions  for  software 
Review  software  acquisition 

Develop  safety-critical  software  tracking  process 
Conduct  Software  Subsystem  Hazard  Analysis 
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Project  /  Software  Conception  Documentation 


Document 

Software  Safety  Section 

Software  Safety  Plan 

Include  software  as  a  subsystem.  Identify  tasks  and 
personnel 

Software  Concepts  Document 

Identify  safety-critical  processes 

Software  Management  Plan 

Discuss  coordination  with  systems  safety  tasks,  flow- 
down  incorporation  of  safety  requirements  and 
applicability  to  safety-critical  software 

Software  Security  Plan 

Determine  security  of  safety-critical  software 

Risk  Management  Plan 

Identify  software  risks,  especially  those  related  to 
safety  and  reliability 

Software  Configuration  Management  Plan 

Identification  and  handling  of  safety-critical 
components 

Software  Verification  and  Validation  Plan 

Discuss  verification  and  validation  of  safety-critical 
components 

Software  Quality  Assurance  Plan 

Identify  quality  assurance  support  to  software  safety 
function,  verification  of  software  safety 
requirements,  and  safety  participation  in  software 
reviews  and  inspections 
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Project  Management 
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Project  Management 


Project  Management  can  be  thought  of  as  the 
establishment  and  maintenance  of  an  environment  that 
enables  work  to  be  performed  efficiently  and  effectively 

To  effectively  manage  and  control  a  project,  the  project 
leader  must  be  able  to: 

-  Identify  the  customer(s) 

-  Define  and  manage  the  requirements 

-  Understand  the  system  that  must  be  built 

-  Establish  the  necessary  project  roles 

-  Understand  the  project  factors  that  must  be  managed 

-  Establish  the  project  management  lifecycle 

-  Choose  a  product  lifecycle 
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Project  Management  -  2 


The  purpose  of  project  planning  is  to  establish  and 
maintain  plans  that  define  project  activities  and  activities  of 
other  relevant  stakeholders  that  affect  the  project  and 
includes: 

-  Estimating 

-  Developing  the  project  plan 

-  Interacting  with  the  relevant  stakeholders 

-  Resolving  conflicts 

The  project  plan  will  need  to  be  revised  as  the  project 
progresses 

-  During  the  project’s  lifecycle  it  will  become  necessary  to 
address  changes  in  requirements,  commitment  changes, 
inaccurate  estimates  and  assumptions,  necessary 
corrective  actions  and  process  improvements 
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Iterative  Nature  of  Planning 


Goal  Statement 


Budget 


Work 


Risks 


Schedule 


Resources 


Estimates 


Project 

Baseline 
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The  Project  Plan 
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Plans  That  Affect  the  Project 


All  plans  that  affect  the  project  should  be  reviewed. 
Candidate  plans  that  affect  project  success  include: 

-  Quality  Assurance  plans 

-  Configuration  Management  plans 

-  Risk  Management  strategy 

-  Integration  strategy 

-  Verification  strategy 

-  Validation  strategy 

-  Product  integration  plans 

-  Software  Safety  plans 

-  Documentation  plans 

-  Stakeholder  involvement 

-  Knowledge  and  skills  development 
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Risk  Management 
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Risk  Management 


Risk  management  is  a  continuous,  forward-looking 
process  that  is  an  important  part  of  project 
management 

Risk  management  should  address  issues  that  could 
endanger  achievement  of  critical  objectives 

A  continuous  risk  management  approach  effectively 
anticipates  and  mitigates  risks  that  may  have  a  critical 
impact  on  a  project 

-  Enables  project  participants  to  share  a  future  view  of 
the  project  and  business 

-  Adds  to  project  members  individually  and  collectively 
owning  the  project 
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Elements  of  Risk  Management 


Risk  Identification 
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Establishing  Risk  Thresholds 
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Hazards  and  Risks 


Hazard  analyses  such  as  the  PHAare  not  primarily 
concerned  with  whether  the  hazard  is  likely  to  occur  or 
not 

All  Hazards  are  bad! 

However  without  unlimited  time  and  money  the 
hazards  must  be  prioritized! 

Risk  is  the  combination  of: 

-  The  probability  that  a  program  will  experience  an 
undesired  event  such  as  a  safety  mishap, 
compromise  of  security  or  system  component  failure 

-  The  consequences,  impact,  or  severity  of  the 
undesired  event  were  it  to  occur 
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Hazards  and  Risks  -  2 


Each  project  or  program  needs  to  define  a  set  of 
“hazard  severity”  levels 

“Relative  risk  terms”  may  be  used  as  long  as  they  are 
accompanied  by  explanations  or  ranges  of  values  that 
contribute  to  a  common  vocabulary  among  project  and 
program  members  when  discussing  hazard  severity 
levels 

Combining  these  two  concepts  (severity  and 
likelihood)  leads  to  a  single  risk  index  value  for  the 
hazard 

-  Risk  index  for  hazards  is  equivalent  to  risk  priority  for 
product  development 
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Hazard  Severity  Definitions 


SIS 

Catastrophic 

Loss  of  human  life  or 

permanent  disability;  loss 
of  entire  systems  of 
ground  facility;  severe 
environmental  damage 

Critical 

Severe  injury  or 
temporary  disability; 
major  system  or 
environmental  damage 

Moderate 

Minor  injury;  minor 
system  damage 

Negligible 

No  injury  or  minor  injury; 
some  system  stress,  but 
no  system  damage 
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Likelihood  of  Occurrence  Definitions 


Likely 

The  event  will  occur 

frequently,  such  as 
greater  than  1  out  of  10 

times 

Probable 

The  event  will  occur 

several  times  in  the  life  of 

an  item 

Possible 

Likely  to  occur  sometime 

in  the  life  of  an  item 

Unlikely 

Remote  possibly  of 
occurrence  during  the  life 

of  an  item 

Improbable 

Very  rare,  possibility  is 
like  winning  the  lottery 
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Hazard  Prioritization 


Severity 

Levels 


Likelihood  of  Occurrence 
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System  Risk  Index 


System  Risk  Index 

Class  of  Safety  Activities  Recommended 

1 

Not  Applicable  as  is  (Prohibited) 

2 

Full 

3 

Moderate 

4,5 

Minimum 

6,7 

None  (Optional) 
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Requirements 

Engineering 
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Requirements 


Requirements  elicitation,  analysis,  review,  traceability  and 
management  are  key  elements  of  a  successful  and  safe 
software  development  process 

Many  of  the  costly  and  critical  system  failures  that  are 
attributed  to  software  can  ultimately  be  traced  back  to 
missing,  incorrect,  misunderstood,  or  incompatible 
requirements 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


Requirements  -  2 


Software  requirements  flow  down  from  many  sources 
including: 

-  System  Requirements 

-  Safety  and  Security  Standards 

-  Hazard  and  Risk  Analyses 

-  System  Constraints 

-  Customer  Input 

-  Software  Safety  “Best  Practices” 
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Requirements  -  3 


The  software  system  must  be  evaluated  to  determine  if 
any  of  it  is  safety-critical  or  has  safety-critical 
characteristics. 

-  Top-down  analyses  such  as  Software  Fault  Tree 
Analysis  are  often  used  to  identify  safety-critical 
software 

-  Any  safety-critical  characteristics  found  during  the 
requirements  analysis  should  be  written  into  the 
software  and  system  requirements 
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Requirements  -  4 


Software  Safety  requirements  are  derived  from  the 
system  and  subsystem  safety  requirements  which 
were  developed  to  mitigate  hazards  identified  in  the 
Preliminary  Hazard  Analysis  (PHA) 

Software  Safety  requirements  should  be  included  in: 

-  Software  Requirements  Specification 

-  Software  Interface  Specification  or  Interface  Control 
Document 

-  Requirements  Traceability  Matrix  [See  Diagrams  on 
Traceability  Below] 
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Early  Phase 

Requirements  Validation 


Customer  requirements  should  be  validated  early  in 
the  development  schedule  to  gain  confidence  that  the 
customer  requirements  are  capable  of  guiding  a 
development  that  results  in  the  customer’s  operational 
needs  being  met 

-  Simulations 

-  Prototypes, 

-  Analyses 

-  Scenarios 

-  Storyboards 

Validation  of  requirements  acts  as  elicitation  of 
requirements  for  the  “next  round” 
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Spiral  Model  of  the  Product 
Requirements  Engineering  Process 
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Bi-Directional  Requirements 
Traceability 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


Customer 
needs, 
expectations 
and  constraints 


jUsed  With  Permission 
£>f  Dr.  Amir  Tomer 
Rafael  -  Israel 


Manage 

Requirement  Changes 

"  s 


istomer  \ 
Requirements 


I 


1  im 

0" 

Product  ^ 

=:  Requirements 

^  i  ^ 

rri 

\ 

Product 
Component 
Requirements 


r 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


Customer 
needs, 
expectations 
and  constraints 


Manage 

Design 

Change 


> 


\ 


jUsed  With  Permission 
£>f  Dr.  Amir  Tomer 
Rafael  -  Israel 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


Development  of  Software 
Safety  Requirements 
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Development  of  Software  Safety 
Requirements 


Software  safety  requirements  are  obtained  from  various 
sources  and  are  usually  classified  as  generic  or  specific 

Generic  Software  Safety  Requirements  -  Generic  software 
safety  requirements  are  derived  from  sets  of  requirements 
and  best  practices  used  in  different  programs  and 
environments  to  solve  common  software  safety  problems 

-  Capture  lessons  learned  and  provide  a  valuable  resource 
for  developers 

-  Prevent  costly  duplication  of  effort  by  taking  advantage  of 
existing  proven  techniques  and  lessons  learned  rather 
than  reinventing  techniques  or  repeating  mistakes 
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Development  of  Software  Safety 
Requirements  -  2 

Specific  Software  Safety  Requirements  -  Specific  software 
safety  requirements  are  system-unique  functional 
capabilities  or  constraints  that  are  identified  in  the  following 
ways: 

-  Method  1  -  Through  top-down  analysis  of  system  design 
requirements  and  specifications 

-  Method  2  -  From  the  Preliminary  Hazard  Analysis  (PHA) 

-  Method  3  -  Through  bottom-up  analysis  of  design  data 
(e.g.,  flow  diagrams,  Failure  Mode  Effects  and  Criticality 
Analysis  (FMECA) 
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Fault  and  Failure  Tolerance 


A  fault  is  an  error  that  does  not  affect  the  functionality 
of  the  system 

-  Bad  data 

-  Unknown  command 

-  Command  or  data  coming  at  an  unknown  time 

A  failure  is  the  inability  of  a  system  or  system 
component  to  perform  its  required  functionality  within 
specified  performance  requirements 

Consequences  of  Failures  and  Faults 

-  A  fault  may  or  may  not  lead  to  a  failure 

-  One  or  more  faults  can  become  a  failure 

-  All  failures  are  the  result  of  one  or  more  faults 
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Fault  Tolerance  -  2 


Fault  Tolerance  (system  failure)  is  the  ability  of  the 
system  to  withstand  an  unwanted  event  and  maintain 
a  safe  and  operational  condition 

-  It  is  determined  by  the  number  of  faults  that  can  occur  in  a 
system  or  subsystem  without  the  occurrence  of  a  failure 

Software  fault  tolerance  is  usually  concerned  with 
detecting  and  recovering  from  small  defects  before 
they  can  become  larger  failures 

-  EXAMPLE:  Error  detection  and  handling  is  one  example  of 
fault-tolerant  software  coding  practices 
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Hazardous  Commands 


A  “Hazardous”  command  is  one  whose  execution  could 
lead  to  an  identified  critical  or  catastrophic  hazard  or  a 
command  whose  execution  can  lead  to  reduced  control  of 
a  hazard 

Commands  can  be  internal  to  a  software  set 

-  From  one  component  to  another 

-  External  crossing  an  interface  to/from  hardware  or  a 
human  operator 

-  Longer  command  paths  increase  the  probability  of  an 
undesired  or  incorrect  command  response  due  to  noise  on 
the  communication  channel,  link  outages,  equipment 
malfunctions  or  human  error 
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Timing,  Sizing,  and  Throughput 
Considerations 


System  design  should  consider  real-world  parameters  and 
constraints  including  human  operator  and  control  system 
response  times,  and  flow  these  down  to  software 

-  Adequate  margins  of  capacity  should  be  provided  for  all  of 
these  critical  resources 

Guidance  for  developers  in  specifying  software 
requirements  to  meet  safety  objectives  are  provided 
here  in  overview  fashion  and  discussed  later  in 
Software  Safety  Requirements  Analysis 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


68 


Timing,  Sizing,  and  Throughput 
Considerations  -  2 


Time  to  Criticality  -  Safety-critical  systems  have  a 
characteristic  “time  to  criticality”  which  is  the  time  interval 
between  a  fault  occurring  and  the  system  reaching  an 
unsafe  state 

-  This  interval  represents  a  time  window  in  which  automatic  or 
manual  recovery  and/or  safing  actions  can  be  performed  either 
by  software,  hardware,  or  by  a  human  operator 

Automatic  Safing  -  Required  if  the  time  to  criticality  is 
shorter  than  the  realistic  human  operator  response  time,  or 
if  there  is  no  human  in  the  loop 
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Timing,  Sizing,  and  Throughput 
Considerations  -  3 


Control  System  Design  -  Control  system  design  defines 
system  timing  requirements 

-  Computerized  control  systems  use  sampled  data  (versus 
continuous  data) 

Sampling  Rates  -  Sampling  rates  should  be  selected  with 
consideration  for  noise  levels  and  expected  variations  of 
control  system  and  physical  parameters 

-  For  measuring  signals  that  are  not  critical,  the  sample  rate 
should  be  at  least  twice  the  maximum  expected  signal 
frequency 

-  For  critical  signals  and  parameters  used  for  a  closed  loop 
control,  the  sampling  rate  is  normally  much  higher 
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Timing,  Sizing,  and  Throughput 
Considerations  -  4 


Dynamic  Memory  Allocation  -  Whether  a  dynamic 
allocation  will  fail  or  succeed  depends  on 

-  How  much  actual  memory  is  available 

-  Whether  virtual  memory  is  to  be  used 

-  How  much  memory  the  software  programs  and  the 
operating  system  use  statically 

-  How  much  memory  is  intended  to  be  dynamically  allocated 

How  the  software  will  deal  with  failed  dynamic  allocation 
should  be  specified 

-  Allowing  a  default  like  “abort,  retry,  fail”  is  a  very  bad  idea 
for  safety-critical  software 
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Formal  Inspections  of  Software  Requirements 


An  Inspection  is  a  formal  verification  technique  in  which 
life-cycle  work  products  are  examined  in  detail  by  a  group 
of  peers  for  the  explicit  purpose  of  detecting  and  identifying 
defects 

The  process  is  led  by  a  Moderator  or  Facilitator  or 
Inspection  Leader  who  is  not  the  author  and  is  impartial  to 
the  life-cycle  work  products  under  review 

The  author  is  not  allowed  to  act  as  the  Moderator 

Defect  resolution  is  mandatory  and  rework  is  formally 
verified 

Defect  data  is  systematically  collected  and  stored  in  an 
inspection  database 

Data  is  analyzed  to  improve  the  product,  the  process  and 
the  effectiveness  of  the  inspection 
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Why  We  Should  Conduct 
Peer  Reviews? 


The  sooner  a  defect  is  found,  the  cheaper  it  is  to  fix 


Source:  Boehm 
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Peer  Reviews  aim  to  minimize  the  number  of  defects 
being  passed  along  from  one  segment  to  the  next,  by 
finding  and  fixing  the  defects  in  the  segment  in  which 
they  were  created. 
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Test  Planning 


Safety  tests  of  the  system  should  also  be  designed  at  the 
time  the  systems  tests  are  defined. 

-  System  tests  are  designed  to  verify  the  functional  aspects 
of  the  software  under  nominal  conditions  as  well  as 
performance,  load,  stress  and  other  tests  that  verify 
acceptable  behavior  in  non-standard  situations 

-  Non-functional  requirements  such  as  the  quality  factors  of 
reliability,  maintainability,  safety,  security  and  usability 
should  also  be  included  in  Systems  Testing 

-  Safety  tests  should  demonstrate  how  the  software  and 
system  meets  the  safety  requirements  in  the  Software 
Requirements  Document 
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Software  Safety 
Requirements  Analysis 
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Software  Safety  Requirements  Analysis 


The  Requirements  Analysis  activities  verify  that  safety 
requirements: 

-  Were  properly  flowed  down  from  the  system  safety 
requirements 

-  Are  correct,  consistent,  and  complete 

-  Have  covered  all  hazards,  even  new  hazards  such  as 
software  functions  that  can  impact  hazard  controls  and 
ways  the  software  can  behave  that  are  unexpected 

-  These  analysis  activities  are  primarily  top  down  analyses 
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Software  Safety  Requirements  Analysis  -  2 


Bottom-up  analysis  of  software  requirements  such  as 
Requirements  Criticality  Analysis  are  performed  to  identify 
possible  hazardous  conditions 

-  Results  in  another  iteration  of  the  PHA  or  Software 
Subsystem  Hazard  Analysis 

-  May  generate  new  software  requirements 
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Software  Safety  Requirements  Analysis  -  3 


Analyses  related  to  the  Software  Requirements  include: 

-  Software  Safety  Requirements  Flow-down  Analysis 

-  Requirements  Criticality  Analysis 

-  Specification  Analysis 

-  Formal  Methods 

-  Timing,  Throughput  and  Sizing  Analysis 

-  Preliminary  Software  Fault  Tree  Analysis 

-  Preliminary  Software  Failure  Modes  and  Effects  Analysis 
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Software  Safety  Requirements  Flow-down 
Analysis 

Generic  safety  requirements  should  be  established  “a  priori”  and 
placed  into  the  system  specification  and  overall  project  design 
specifications 

-  They  then  should  be  flowed  into  the  subsystem  specifications, 
such  as  the  software  subsystem  requirements 

-  Top-down  analysis 

Other  safety  requirements  are  derived  from  bottom-up  analysis 

-  They  are  flowed  up  from  subsystems  and  components  back  to 
the  system  level  requirements 

-  These  new  system  level  requirements  are  then  flowed  back 
down  across  all  affected  subsystems 

Problems  in  the  flow-down  process  can  be  caused  by: 

-  Incomplete  analysis 

-  Inconsistent  analysis  of  highly  complex  systems 

-  Use  of  ad  hoc  techniques  by  inexperienced  analysts 
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Requirements  Criticality  Analysis 


Criticality  analysis  identifies  program  requirements  that 
have  safety  implications 

-  Analyze  the  hazards  of  the  software  /  hardware  system 
and  identify  those  that  could  present  catastrophic  or  critical 
hazards 

•  This  evaluates  each  program  requirement  in  terms  of  safety 
objectives  derived  for  the  software  component 

-  For  those  requirements  that  have  safety  implications, 
designate  the  requirement  as  “safety-critical” 

-  Place  the  safety-critical  requirement  into  a  tracking  system 
to  ensure  traceability  of  the  software  safety  requirements 
throughout  the  software  development  cycle 

•  From  the  highest  level  specification  all  the  way  to  the  code 
and  test  documentation 
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Requirements  Criticality  Analysis  -  2 


The  System  Safety  organization  should  coordinate  with  the 
Project  System  Engineering  organization  to  review  and 
agree  on  the  criticality  designations 

-  Software  Safety  Engineers  and  Software  Development 
Engineers  should  be  included  in  the  discussion 

-  The  “software”  viewpoint  must  be  part  of  any  systems 
engineering  activity 

-  Requirements  may  be  consolidated  to  reduce  the  number 
of  critical  requirements  or  flagged  for  special  attention 
during  design  to  reduce  the  criticality  level 

-  It  is  always  better  to  remove  or  reduce  requirements  later 
than  to  try  to  add  them  in  when  they  did  not  exist  before 


NATIONAL  RENEWABLE  ENERGY  LABORATORY 


83 


Requirements  Criticality  Analysis  -  3 


Requirements  Criticality  Analysis  Steps: 

-  Analyze  all  software  requirements  to  identify  additional 
potential  system  hazards  that  the  system  PHAdid  not 
reveal 

-  Use  a  checklist  to  identify  PHA-designated  hazards  that 
are  not  reflected  in  the  software  requirements  and  new 
hazards  missed  by  the  PHA 

-  Look  for  areas  where  system  requirements  were  not 
correctly  flowed  to  the  software 

-  Flow  potential  hazards  added  to  the  system  requirements 
down  to  the  subsystems  (hardware,  software  and 
operations) 

-  Review  the  system  requirements  to  identify  hardware  or 
software  functions  that  receive,  pass,  or  initiate  critical 
signals  or  hazardous  commands 
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Requirements  Criticality  Analysis  -  4 


-  Review  the  software  requirements  to  verify  that  the 
functions  from  the  system  requirements  are  included 

-  Look  for  any  new  software  functions  or  objects  that 
receive/pass/initiate  critical  signals  or  hazardous 
commands 

-  Look  through  the  software  requirements  for  conditions  that 
may  lead  to  unsafe  situations 

•  Out  of  sequence 

•  Wrong  event 

•  Inappropriate  magnitude 

•  Incorrect  polarity 

•  Inadvertent  command 

•  Adverse  environment 

•  Deadlocking 

•  Failure-to-command  modes 
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Requirements  Criticality  Analysis  -  5 


Software  safety  analysis  must  also  examine  the 
characteristics  of  the  software  system 

-  All  characteristics  of  safety-critical  software  must  be 
evaluated  to  determine  if  they  are  safety-critical  [See  list  of 
characteristics  -  next  slide] 

-  Safety-critical  characteristics  should  be  controlled  by 
requirements  that  receive  rigorous  quality  control  in 
conjunction  with  rigorous  analysis  and  test 
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Requirements  Criticality  Analysis  -  6 


Safety-critical  characteristics  to  be  considered  include: 

-  Specific  limit  ranges 

-  Out  of  sequence  event  protection  requirements 

-  Timing 

-  Relationship  to  logic  for  limits  (EXAMPLE:  Temperature 
may  be  more  constrained  during  an  experiment  run  than 
when  the  system  is  idle) 

-  Voting  logic 

-  Hazardous  command  processing  requirements 

-  Fault  response 

-  Fault  detection,  isolation,  and  recovery 

-  Redundancy  management  /  switchover  logic 

•  What  components  to  switch,  and  under  what  circumstances 
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Requirements  Criticality  Analysis  -  7 


Resources  that  serve  as  input  to  Requirements  Criticality 
Analysis 

-  Software  Development  Plan 

-  Software  Quality  Assurance  Plan 

-  Software  Configuration  Management  Plan 

-  Risk  Management  Plan 

-  System  Requirements 

-  System  Design  Document 

-  Software  Requirements  Specification 

-  External  Interface  Specification 

-  Functional  Flow  Diagrams 

-  Information  from  the  system  PHA  on  hazardous  sources  that 
may  be  controlled  directly  or  indirectly  by  software 
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Specification  Analysis 


Specification  analysis  evaluates  the  completeness, 
correctness,  consistency,  and  testability  of  software 
requirements. 

-  Well-defined  requirements  are  strong  standards  by  which  to 
evaluate  a  software  component 

Specification  analysis  should  evaluate  requirements 
individually  and  as  an  integrated  set. 

Techniques  to  perform  specification  analysis  include: 

-  Reading  Analysis 

-  Traceability  Analysis 

-  Control-flow  Analysis 

-  Information-flow  Analysis 

-  Functional  Simulation 
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Specification  Analysis  -  2 


The  Safety  Organization  should  ensure  the  software 
requirements  appropriately  influence  the  software  design 
and  the  development  of  the  operator,  user,  and  diagnostic 
manuals. 

The  Safety  Representative  normally  reviews: 

-  System  Specification 

-  Software  Requirements  Specification 

-  Interface  Requirements  Specification 

-  Functional  Flow  Diagrams 

-  Storage  allocation  and  program  structure  documents 

-  Information  concerning  system  energy  and  other  hazardous 
event  sources  that  may  be  controlled  directly  or  indirectly  by 
software 

-  Software  Development  Plan,  Software  Configuration 
Management  Plan,  Software  Quality  Assurance  Plan  and 
Historical  Data 
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Formal  Methods 


Formal  methods  represents  the  process  of  writing  the 
requirements  (specification)  in  a  formal,  mathematical 
language 

-  Even  if  Formal  Verification  is  not  used  to  verify  the  specifications, 
the  act  of  creating  a  Formal  Specification  often  catches  many 
errors 

Formal  Specification  removes  ambiguity  and  uncertainty 

-  It  allows  errors  or  omission  to  be  discovered  including 
undocumented  assumptions  and  inadequate  off-nominal 
behavior 

-  Conflicting  requirements  and  logic  errors  are  also  uncovered 

-  Defects  found  and  corrected  early  in  the  lifecycle  are  much  less 
costly  to  fix 
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Formal  Methods  -  2 


“Model  Checking”  is  a  form  of  Formal  Methods  that  verifies 
finite-state  systems 

-  Model  Checking  is  now  used  by  many  projects  as  an 
analysis  and  verification  technique 

-  Model  Checking  is  especially  oriented  at  the  verification  of 
reactive,  embedded  systems  or  systems  that  are  in 
constant  interaction  with  the  environment 

-  Model  Checking  can  be  easily  applied  at  any  stage  of  the 
existing  software  process  without  causing  major 
disruptions 
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Timing,  Throughput  and  Sizing  Analysis 


Timing,  throughput,  and  sizing  analysis  for  safety-critical 
functions  evaluates  software  requirements  that  relate  to 
execution  time,  I/O  data  rates  and  memory/storage 
allocations 

-  The  analysis  focuses  on  program  constraints 

Typical  constraints  include: 

-  Maximum  execution  time 

-  Maximum  memory  usage 

-  Maximum  storage  size  for  programs 

-  I/O  data  rates  the  program  must  support 

The  Safety  Organization  should  evaluate  the  adequacy  and 
feasibility  of  safety-critical  timing,  throughput,  and  sizing 
requirements 

-  I/O  Channels  may  be  overloaded  by  many  error  messages, 
preventing  safety-critical  features  from  operating 
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Timing,  Throughput  and  Sizing  Analysis  -  2 


Factors  to  consider  for  quantifying  timing/sizing  resource 
requirements: 

-  Memory  Usage  Versus  Availability 

•  Dynamic  Memory  Allocation  -  Can  lead  to  problems 
from  not  freeing  allocated  memory,  freeing  memory 
twice  or  buffer  overruns  that  overwrite  code  or  other 
data  areas 

•  When  data  structures  are  dynamically  allocated,  they 
often  cannot  be  statically  analyzed  to  verify  that  arrays, 
strings,  etc.,  do  not  go  past  the  physical  end  of  the 
structure 
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Timing,  Throughput  and  Sizing  Analysis  -  3 


-  I/O  Channel  Usage  (Load)  Versus  Capacity  and 
Availability 

•  Look  at  the  amount  of  input  data  (science  data, 
housekeeping  data,  control  sensors)  and  the  amount  of 
output  data  (communications)  generated 

•  I/O  Channel  should  include: 

-  Internal  hardware  (sensors), 

-  Interprocess  communications  (messages) 

-  External  communications  (data  output,  command,  and 
telemetry  interfaces) 
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Timing,  Throughput  and  Sizing  Analysis  -  4 


-  Execution  Times  Versus  CPU  Load  and  Availability 

•  Investigate  the  time  variations  of  CPU  load  and 
determine  the  circumstances  that  generate  peak  load 

•  Is  the  execution  under  high  load  conditions  acceptable? 

•  Consider  the  timing  effects  from  multitasking 

-  Message  passing  delays 

-  Inability  to  access  a  needed  resources  because  another 
task  has  it 

-  Excessive  multitasking  can  result  in  a  system  instability 
leading  to  “crashes” 
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Timing,  Throughput  and  Sizing  Analysis  -  5 


-  Sampling  Rates  Versus  Rates  of  Change  of  Physical 
Parameters 

•  Analysis  should  address  the  validity  of  the  system 
performance  models  used,  together  with  simulation  and 
test  data 

-  Program  Storage  Space  Versus  Executable  Code 
Size 

•  Estimate  the  size  of  the  executable  software  in  the 
device  it  is  stored  in 

•  The  program  size  includes  the  operating  system  as  well 
as  the  application  software 
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Timing,  Throughput  and  Sizing  Analysis  -  6 


-  Amount  of  Data  to  Store  Versus  Available  Capacity 

•  Consider  how  much  science,  housekeeping,  or  other 
data  will  be  generated  and  the  amount  of  storage 
space  available 

•  Under  some  conditions,  being  unable  to  save  data  or 
overwriting  previous  data  could  be  a  safety  related 
problem 
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Software  Fault  Tree  Analysis 


It  is  possible  for  a  system  to  meet  requirements  for  a 
correct  state  and  still  be  unsafe 

-  Complex  systems  increase  the  chance  that  unsafe  modes 
will  not  be  caught  until  the  system  is  in  the  field 

-  Fault  Tree  Analysis  (FTA)  is  one  method  that  focuses  on 
how  errors  can  lead  to  hazards 

-  Software  Fault  Tree  Analysis  (SFTA)  is  an  extension  of  the 
hardware  FTA  into  the  software  arena 

-  The  requirements  phase  is  the  time  to  perform  a 
preliminary  SFTA 

•  The  Preliminary  Hazard  Analysis  or  Software  Subsystem 
Hazard  Analysis  is  the  primary  source  for  hazards  along  with 
the  Requirements  Criticality  Analysis 

-  The  result  of  a  fault  tree  analysis  is  a  list  of  contributing 
causes  (e.g.,  states  or  events)  that  can  lead  to  a  hazard 
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Software  Fault  Tree  Analysis  -  2 


Software  Fault  Tree  Analysis  (SFTA)  works  well  in 
combination  with  the  Software  Failure  Modes  and  Effects 
Analysis  (SFMEA) 

-  The  SFTA  is  top-down  working  from  the  hazard  to  possible 
causes 

-  The  SFMEA  starts  with  individual  components  and  works 
from  the  bottom  (component)  up  to  a  hazard 

-  When  used  together,  these  two  analyses  are  very  good  at 
finding  all  of  the  possible  failure  modes  or  areas  of 
concern 
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Software  Failure  Modes  and  Effects  Analysis 


Failure  Modes  and  Effects  Analysis  is  a  bottom-up 
technique  that  looks  at: 

-  How  each  component  could  fail 

-  How  the  failure  propagates  through  the  system 

-  Whether  it  can  lead  to  a  hazard  or  not 

-  A  detailed  design  is  required  for  an  effective  FMEAto  be 
conducted 

Software  Failure  Modes  and  Effects  Analysis  (SFMEA) 
uses  the  methods  of  a  standard  (hardware)  FMEA, 
substituting  software  components  for  hardware 
components  in  each  case 
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Software  Failure  Modes  and  Effects  Analysis  -  2 


There  are  several  types  of  FMEAs: 

-  System  -  focuses  on  global  system  functions 

-  Design  -  focuses  on  components  and  subsystems 

-  Process  -  focuses  on  manufacturing  and  assembly 
processes 

-  Service  -  focuses  on  service  functions 

-  Software  -  focuses  on  software  functions 
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Software  Failure  Modes  and  Effects  Analysis  -  3 


Benefits  of  FMEA 

-  Provide  the  engineer  with  a  tool  that  can  assist  in  providing 
reliable,  safe,  and  customer  pleasing  products  and  processes 

-  Improve  product/process  reliability  and  quality 

-  Increase  customer  satisfaction 

-  Early  identification  and  elimination  of  potential  product/process 
failure  modes 

-  Prioritize  product/process  deficiencies 

-  Capture  engineering/organization  knowledge 

-  Emphasizes  problem  prevention 

-  Documents  risk  and  actions  taken  to  reduce  risk 

-  Provide  focus  for  improved  testing  and  development 

-  Minimizes  late  changes  and  associated  cost 

-  Catalyst  for  teamwork  and  idea  exchange  between  functions 
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Software  Failure  Modes  and  Effects  Analysis  -  4 


FMEA  Procedure  -  Overview 

-  1 .  Describe  the  product/process  and  its  function  -  It  is 
important  to  consider  both  intentional  and  unintentional 
uses  since  product  failure  often  ends  in  litigation,  which 
can  be  costly  and  time  consuming. 

-  2.  Create  a  Block  Diagram  of  the  product  or  process  -  The 
diagram  shows  the  logical  relationships  of  components 
and  establishes  a  structure  around  which  the  FMEA  can 
be  developed 

-  3.  Complete  the  header  on  the  FMEA  Form  worksheet. 
(See  Example) 

-  4.  Use  the  worksheet  to  begin  listing  items  or  functions 
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EXAMPLE  FMEA  -  Spreadsheet 


Part/Product  Name  and  No.:  Part/Product  Owner:  Prepared  By: 

Process  Name:  Process  Owner:  FMEA  Date  (Orig.):  (Rev.): 

Other  Areas  Involved: 


Item  / 
Function 

Potential 

Failure 

Mode 

Potential 

Failure 

Effects 

S 

E 

V 

Potential 

Causes 

O  o  o 

Current 

Design 

Controls 

D 

E 

T 

R 

P 

N 

Recommended 

Actions 

Actions 

or 

Plans 

P 

S 

P 

0 

P 

D 

P 

R 

P 

N 
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Software  Failure  Modes  and  Effects  Analysis  -  5 


-  5.  Identify  Failure  Modes  -  A  failure  mode  is  defined  as  the 
manner  in  which  a  component,  subsystem,  system, 
process,  etc.  could  potentially  fail  to  meet  the  design 
intent.  Examples  of  potential  failures  modes  include: 

•  Corrosion 

•  Hydrogen  embrittlement 

•  Electrical  Short  or  Open 

•  Torque  Fatigue 

•  Deformation 

•  Cracking 

-  6.  List  failure  modes  for  function  of  each  component  or 
process  step  -  a  failure  mode  in  one  component  can  serve 
as  the  cause  of  a  failure  mode  in  another  component 
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Software  Failure  Modes  and  Effects  Analysis  -  6 


-  7.  Describe  the  effects  of  those  failure  modes  -  A  failure 
effect  is  defined  as  the  result  of  a  failure  mode  on  the 
function  of  the  product/process  as  perceived  by  the 
customer.  Examples  of  failure  effects  include: 

•  Injury  to  the  user 

•  Inoperability  of  the  product  or  process 

•  Improper  appearance  of  the  product  or  process 

•  Odors 

•  Degraded  performance 

•  Noise 
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Software  Failure  Modes  and  Effects  Analysis  -  7 


-  8.  Identify  the  causes  for  each  failure  mode  -  A  failure 
cause  is  defined  as  a  design  weakness  that  may  result  in  a 
failure.  Examples  of  potential  causes  include: 

•  Improper  torque  applied 

•  Improper  operating  conditions 

•  Contamination 

•  Erroneous  algorithms 

•  Improper  alignment 

•  Excessive  loading 

•  Excessive  voltage 
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Software  Failure  Modes  and  Effects  Analysis  -  8 


-  9.  Enter  the  Probability  Factor -A  numerical  weight  should 
be  assigned  to  each  cause  that  indicates  how  likely  that 
cause  is  to  occur 

-  10.  Identify  Current  Controls  (design  or  process)  -  Current 
controls  (design  or  process)  are  the  mechanisms  that 
prevent  the  cause  of  the  failure  mode  from  occurring  or 
which  detect  the  failure  before  it  reaches  the  customer. 

•  Controls  should  be  assessed  to  determine  how  well  it  is 
expected  to  identify  or  detect  failure  modes 

•  Previously  undetected  or  unidentified  failure  modes  should 
be  used  to  update  the  FMEA 
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Software  Failure  Modes  and  Effects  Analysis  -  9 


-  11 .  Determine  the  likelihood  of  Detection  -  Detection  is  an 
assessment  of  the  likelihood  that  the  Current  Controls  will 
detect  the  Cause  of  the  Failure  Mode  or  the  Failure  Mode 
itself  thus  preventing  it  from  reaching  the  Customer 

-  12.  Review  Risk  Priority  Numbers  -  The  Risk  Priority 
Number  is  a  mathematical  product  -  RPN  =  (Severity)  x 
(Probability)  x  (Detection) 
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Software  Failure  Modes  and  Effects  Analysis  - 10 


-  13.  Determine  Recommended  Actions  to  address  potential 
failures  that  have  a  high  RPN  -  These  actions  could 
include: 

•  Specific  inspection,  testing,  or  quality  procedures 

•  Selection  of  different  components  or  materials 

•  Limiting  environmental  stresses  or  operating  range 

•  Re-design  of  the  item  to  avoid  the  failure  mode 

•  Performing  preventative  maintenance 

•  Inclusion  of  back-up  systems  or  redundancy 

-  14.  Assign  Responsibility  and  a  Target  Completion  Date  for 
these  actions  -  Make  responsibility  clear  cut! 
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Software  Failure  Modes  and  Effects  Analysis  - 11 


-  1 5.  Indicate  Actions  Taken  -  After  the  actions  have  been 
taken,  re-assess  the  severity,  probability  and  detection  and 
review  the  revised  RPNs 

•  Are  further  actions  needed? 

-  16.  Update  the  FMEA  as  the  design  or  process  changes 
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Software  Requirements  Tasks 


Software  Engineering  Tasks 

System  and  Software  Safety 

Tasks 

Software  Requirements  Development 

Development  of  software  safety-critical 
requirements 

Requirements  Management 

Safety  Requirements  Flow-down  Analysis 

Formal  Methods  for  Specification 

Requirements  Criticality  Analysis 

Formal  Inspection  of  Software 
Requirements 

Specification  Analysis  of  Safety-Critical 
Requirements 

System  Test  Planning 

Software  Fault  Tree  Analysis 

Timing,  Throughput  and  Sizing 
Considerations 

Software  Failure  Modes  and  Effects 
Analysis 

Formal  Inspections  of  Software 
Requirements 
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Software  Requirements  Documentation 


Document 

Software  Safety  Section 

Software  Requirements  Document 

Identification  of  all  safety-critical  software 
requirements 

Software  Interface  Specification 

Identification  of  any  interfaces  that  are  part  of 
safety-critical  elements 

Formal  Inspection  of  Software  Requirements 

Identification  of  any  safety-critical  requirements 
defects  that  are  considered  major  (must  be 
fixed) 

Requirements  Traceability  Matrix 

Special  identification  given  to  safety-critical 
requirements 

Analysis  Reports 

Identification  of  any  safety-related  aspects  or 
safety  concerns 

Acceptance  Test  Plan 

This  is  the  customer  acceptance  test.  Includes 
all  safety  testing  necessary  to  assure  the 
customers  that  the  system  is  safe 

Systems  Test  Plan 

Includes  stress,  load,  disaster,  stability,  and 
other  tests  including  functional  testing. 

Verifies  that  the  system  cannot  go  into  an 
unsafe  mode  under  adverse  conditions 
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Software  Configuration 
Management 
and 

Software  Quality 
Assurance 
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Software  Configuration  Management 


Software  Configuration  Management  (SCM)  is  a 
critical  Project  Management  support  function 

-  One  cannot  product  “safe”  software  without  software 
configuration  management 

-  The  system  or  system  component  cannot  be  declared 
“quality”  if  there  is  no  control  over  the  pieces  that 
have  been  developed 

-  SCM  is  a  process  to  maintain  and  monitor  the 
software  development  process 
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Software  Configuration  Management  -  2 


Software  Configuration  Management  includes: 

-  Identification 

•  Identifying  the  structure  and  kinds  of  components 

•  Making  them  unique  and  accessible  in  some  form  by 
giving  each  component  a  name,  version  identification 
and  configuration  identification 

-  Control 

•  Controlling  the  release  of  a  product  and  changes  to  it 
throughout  the  project/product  lifecycle  by  having 
controls  in  place  that  ensure  consistent  software  via  the 
creation  of  a  baseline  product 
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Software  Configuration  Management  -  3 


-  Status  Accounting 

•  Recording  and  reporting  the  status  of  components  and 
change  requests 

•  Gathering  vital  statistics  about  components  in  the 
product 

-  Configuration  Auditing 

•  Validating  the  completeness  of  a  product 

•  Maintaining  consistency  among  the  components  by 
ensuring  that  the  components  are  reviewed  for  possible 
change  when  the  requirements  they  are  associated 
with  are  changed 
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Software  Configuration  Management  -  4 


-  Change  Control 

•  Change  control  is  an  important  part  of  developing  safe 
software 

•  Arbitrary  changes  should  be  avoided 

•  Once  a  piece  of  software  has  reached  a  level  of 
maturity,  any  subsequent  change  request  should  go 
through  a  formal  change  control  process 

•  Changes  to  the  lifecycle  work  products,  especially  the 
code  should  be  accompanied  by  comments  that 
indicate 

-  Why  the  change  occurred 

-  What  was  changed 

-  When  it  was  changed 

-  Who  changed  it 
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Software  Configuration  Management  -  5 


-  Defect  Tracking 

•  Record  all  the  defects  for  future  reference 

•  Having  defect  information  from  previous  projects  can 
be  a  big  plus  when  debugging  the  next  project 

•  Recording  the  defects  allows  metrics  to  be  determined 

-  One  of  the  easiest  ways  to  judge  whether  a  program  is 
ready  for  serious  safety  testing  is  to  measure  its  defect 
density,  the  number  of  defects  per  line  of  code 
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Software  Quality  Assurance 


Software  Quality  Assurance  should  begin  in  the  early  phases  of 
a  project  to  establish  plans,  processes,  standards,  and 
procedures  that  will: 

-  Add  value  to  the  project 

-  Satisfy  the  requirements  of  the  project  and  organizational 
policies 

SQA  Representatives  should  participate  in  the  establishment  of 
those  plans,  processes,  standards,  and  procedures  to  ensure 
they  will  fit  or  can  be  tailored  to  fit  the  project’s  needs  and  that 
they  will  be  usable  for  performing  quality  assurance  evaluations 

SQA  Representatives  provides  project  staff  and  managers  at  all 
levels  with  appropriate  visibility  into,  and  feedback  on,  processes 
and  associated  work  products  throughout  the  life  of  the  project 
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Software  Quality  Control  vs. 
Software  Quality  Assurance 


Quality  Control  evaluates  the  products 

-  Checks  to  see  if  the  requirements  for  the  product  or 
product  components  are  satisfied  (Verification) 

-  Checks  the  quality  of  the  product(s) 

•  Is  the  product  within  tolerance? 

•  Is  the  product  (or  life-cycle  work  product)  of  an  acceptable 
quality? 

-  Tools  and  Techniques 

•  reviews 

•  tests 

•  inspections 
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Software  Quality  Control  vs. 
Software  Quality  Assurance  -  2 


Quality  Assurance  evaluates  the  process 

-  Checks  that  the  process  is  working 

•  Is  the  process  being  followed? 

•  Are  the  QC  checks  being  applied? 

•  Are  the  QC  checks  efficient? 

•  Is  the  process  causing  quality  problems? 

•  Is  the  process  working  for  the  organization? 

-  Tools  and  Techniques 

•  Audits  /  Objective  Evaluations 

•  Assessments  /  Appraisals 
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Software  Quality  Reports 

to  Management _ 

Software  Quality  Reporting  should  include  comments  about 
the  processes,  standards,  etc.,  to  the  following  questions: 

-  Are  they  efficient? 

-  Are  they  effective? 

-  Does  following  these  processes  result  in  the  necessary 
product  quality? 

-  If  Software  Safety  is  required,  the  quality  report  must 
indicate  if  the  processes  are  being  followed  and  are 
adequate  to  produce  “safe  software” 

Quality  reports  are  sent  to  Higher-level  Management  for 
their  analysis  and  action 
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Safety  as  a 
Quality  Factor 

Concepts  Taken  from 
“Common  Concepts  Underlying  Safety, 
Security,  and  Survivability 
Engineering” 

Donald  G.  Firesmith 
December  2003 
CMU/SEI-2003-TN-033 
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Analysis  of  Software  Intensive  Systems 


Software-intensive  systems  are  commonplace,  and  society 
relies  heavily  upon  them 

-  Software  is  found  in  automobiles,  airplanes,  chemical 
factories,  power  stations  and  other  mission  critical  systems 

-  Software-intensive  systems  are  neither  perfect  nor 
invulnerable 

•  They  fail  due  to  software  defects,  hardware  breakdowns, 
accidental  misuse  and  deliberate  abuse 
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126 


Analysis  of  Software  Intensive  Systems  -  2 


Safety,  security,  and  survivability  engineering  are  three  very 
close  related  disciplines 

-  Analysis  of  all  three  is  inherently  difficult  -  these  requirements 
specify  what  is  to  be  prevented  such  as  attacks  and  safety 
hazards  rather  than  a  desired  capability 

-  They  deal  with  assets  that  must  be  protected  and  with  the  risks 
of  harm  to  these  assets  that  must  be  managed 

-  There  is  an  inherent  level  of  uncertainty  because  what  these 
requirements  seek  to  prevent  may  or  may  not  ever  happen! 

-  This  is  especially  true  of  safety  requirements! 

-  Hazards  and  threats  associated  with  software-intensive  systems 
are  also  constantly  changing,  making  the  risks  very  difficult  to 
quantify 
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Analysis  of  Software  Intensive  Systems  -  3 


Quality  means  much  more  than  merely  meeting  functional 
requirements 

-  An  application  that  meets  all  of  its  required  features  and 
fulfills  each  of  its  use  cases  can  still  be  totally 
unacceptable 

•  Inadequate  availability 

•  Capacity  too  low 

•  Performance  too  slow 

•  Not  interoperable  with  other  systems 

•  Not  safe  to  use 

•  Numerous  security  vulnerabilities 

•  Not  considered  to  be  user  friendly 

It  is  not  adequate  to  specify  required  quality  by  stating 
that  an  application  shall  have  high  capacity  and  be 
safe,  secure  and  usable 
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Analysis  of  Software  Intensive  Systems  -  4 


Quality  Model 

-  A  quality  model  decomposes  quality  into  its  component 
quality  factors 

•  Attributes 

•  Subfactors 

-  It  provides  specific  quality  criteria  (descriptions) 

-  It  provides  metrics  or  means  of  measurement 

•  Turns  general  high-level  quality  factors  into  detailed  and 
specific  measureable  descriptions 

-  It  describes  the  capabilities  of  a  systems  beyond  its 
functionality 
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Analysis  of  Software  Intensive  Systems  -  5 


Quality  Model  -  a  hierarchical  model  for  formalizing  the 
concept  of  quality  in  terms  of  its  component  quality  factors 
and  subfactors 

Quality  Factor  (Quality  Attribute)  -  a  high-level  characteristic  or 
attribute  of  something  that  captures  an  aspect  of  its  quality 

-  Quality  has  to  do  with  the  degree  to  which  something 
possesses  a  combination  of  characteristics,  attributes, 
aspects,  or  traits  that  are  desirable  to  its  stakeholders 

-  Quality  Factors  include  availability,  expandability, 
maintainability,  reliability,  reusability,  safety,  security  and 
usability 
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Analysis  of  Software  Intensive  Systems  -  6 


Quality  Subfactor  -  a  major  component  (aggregation)  of  a 
quality  factor  or  another  quality  subfactor 

Quality  Criterion  -  a  specific  description  of  something  that 
provides  evidence  either  for  or  against  the  existence  of  a  specific 
quality  factor  or  subfactor 

Quality  Metric  -  a  metric  that  quantifies  a  quality  criterion  and 
thus  makes  it  measurable,  objective,  and  unambiguous 

System  -  an  integrated  collection  of  data  components,  hardware 
components,  software  components,  human-role  components 
and  document  components  that  collaborate  to  provide  some 
cohesive  set  of  functionality  with  specific  levels  of  quality 
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Model  for  a  Quality  Model 
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Safety  as  a  Quality  Factor 


Safety  -  as  a  Quality  Factor,  safety  can  be  classified  into 
subclasses: 

-  Health  Safety 

-  Property  Safety 

-  Environmental  Safety 
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Safety  as  a  Quality  Factor  -  2 


Safety  -  the  degree  to  which  accidental  harm  is  prevented, 
detected,  and  properly  reacted  to 

-  People  are  not  considered  safe  unless  they  are  safe  from 
both  accidental  and  malicious  harm 

Health  Safety  -  the  degree  to  which  illness,  injury,  and 
death  are  prevented,  detected,  and  properly  reacted  to 

-  Health  safety  involves  all  people  who  may  reasonably  be 
expected  to  be  harmed  by  the  system  during  an  accident 

Property  Safety  -  the  degree  to  which  accidental  damage 
and  destruction  or  property  is  prevented,  detected,  and 
properly  reacted  to 

Environmental  Safety  -  the  degree  to  which  accidental 
damage  to  and  destruction  of  parts  of  the  environment  is 
prevented,  detected,  and  properly  reacted  to 
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Tailoring  the  Software 
Safety  Effort 


Once  the  scope  of  the  software  safety  effort  has  been 
determined,  the  project  or  program  processes  can  be 
tailored 

The  safety  activities  should  be  sufficient  to  match  the 
software  development  effort  and  yet  ensure  that  the 
overall  system  will  be  safe! 
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Tailoring  the  Software 
Safety  Effort  -  2 


Development 

The  use  of  safety  features  such  as  firewalls  depends 
on  where  it  is  best  applied  and  needed.  The  degree 
to  which  each  of  these  activities  is  performed  is 
related  to  the  software  risk.  Software  Safety  features 
should  be  reflected  in  the  requirements 

Analysis 

There  are  many  types  of  analyses  that  can  be 
completed  during  software  development.  The 
analyses  can  range  from  Requirements  Criticality 
Analysis  to  Software  Fault  Tree  Analysis  of  the  design 

to  Formal  Methods 
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Tailoring  the  Software 
Safety  Effort  -  3 


Inspections 

Inspections  can  take  place  in  a  number  of  settings  and  with 
varying  products  (requirements  to  test  plans).  The  number 
of  inspections  and  products  is  dependent  on  the  risk 
related  to  the  system.  Inspections  is  a  form  of  Peer 

Reviews  that  can  include  Structured  Walkthroughs, 
Walkthroughs,  Circulation  Reviews,  and  Buddy  Checks. 
Technical  Reviews  that  involves  a  group  of  complementary 
disciplines  to  seek  alternative  solutions  is  a  Review  but  not 
normally  considered  a  Peer  Review. 

Reviews 

The  number  of  formal  reviews  and  the  setting  up  of  delta 
or  follow-up  reviews  can  be  used  to  give  the  organization 
more  places  to  look  at  the  products  as  they  are  being 
developed. 
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Tailoring  the  Software 
Safety  Effort  -  4 


Verification 

Activities 


Verification  includes  verification  of  the  product  and  intermediate  work  products 
against  selected  requirements,  including  customer,  product,  and  product 
component  requirements.  Verification  is  inherently  an  incremental  process.  It 
begins  with  verification  of  the  requirements  and  progresses  through  the  verification 
of  the  evolving  work  products.  Verification  culminates  in  the  verification  of  the 
completed  product  normally  in  Systems  Test  although  sometimes  Acceptance 
Testing  serves  as  the  final  Verification  activity. 

Verification  methods  should  address  re-verification  to  ensure  that  rework 
performed  on  work  products  do  not  cause  unintended  defects.  Methods  of 
verification  include  but  are  not  limited  to: 

Inspections  and  Structured  Walkthroughs 
Audits 

Path  coverage  testing 
Simulations 
Modeling 
Functional  testing 
Decision  table-based  testing 
Load,  stress,  and  performance  testing 
Operational  scenario  testing 
Observations  and  demonstrations 

Verification  and  Validation  environments  may  be  built,  reused,  purchased,  or 
shared  or  some  combination  of  these. 
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Tailoring  the  Software 
Safety  Effort  -  5 


Validation  demonstrates  that  a  product  or  product  component 
fulfills  its  intended  use  when  placed  in  its  intended  environment 
(operational  environment)  by  the  intended  end  users  who  must 
use  the  system  in  that  operational  environment. 

Validation  methods  are  selected  based  on  their  ability  to 
demonstrate  that  user  needs  are  satisfied.  To  support 
validation,  the  requirements  must  be  defined  for  a  realistic 
environment  including: 

Functionality  (implementation) 

Maintenance 
Sustainment 
Reuse 
Installation 
Training 
Support 
Performance 
Quality  Attributes 
Disposal  as  appropriate 
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Verification  and  Validation  Tailoring 


Verification  and  Validation  activities  may  be  tailored  by 
determining  the  types  of  testing  based  on  the  software 
risk,  complexity  and  criticality  of  the  software  product  or 
product  component 

Unit  Testing  may  be  performed  by  the  software  developer, 
by  a  peer  developer  or  by  an  outside  supplier  depending 
on  those  factors,  the  mission  and  the  expectations  of  the 
customer  as  well  as  the  end  users  and  the  operational 
environment  the  product  must  operate  in 

Maintenance  and  expansion  constraints  or  requirements, 
support,  training,  usability  and  reliability  requirements  all 
may  have  an  effect  on  the  tailoring  of  the  Verification  and 
Validation  activities 
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“Full”  Software  Safety  Effort 


Systems  and  subsystems  that  have  severe  hazards 
which  can  escalate  to  major  failures  in  a  very  short 
period  of  time  require  the  greatest  level  of  software 
safety  effort 

-  These  systems  may  require  a  formal,  rigorous  program  of 
quality  and  safety  assurance  to  ensure  complete  coverage 
and  analysis  of  all  requirements,  design,  code,  and  tests 

-  Safety  analyses,  software  development  analyses,  safety 
design  features,  and  Software  Quality  Assurance 
oversight  are  highly  recommended 
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“Moderate”  Software 
Safety  Effort 

Systems  and  subsystems  which  fall  into  this  category 
typically  have  either 

-  1)A  limited  hazard  potential 

-  2)The  response  time  for  initiating  hazard  controls  to 
prevent  failures  is  long  enough  to  allow  for  human 
operators  to  respond  to  the  hazardous  situation 

These  systems  require  a  rigorous  program  for  safety 
assurance  of  software  identified  as  safety-critical 

-  Some  analyses  are  required  to  assure  there  are  no 
“undiscovered”  safety-critical  areas  that  may  need 
software  safety  features 
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“Minimum”  Software 
Safety  Effort 


For  systems  in  this  category,  either  the  inherent  hazard 
potential  of  a  system  is  very  low  or  control  of  the 
hazard  is  accomplished  by  non-software  means 

-  Software  development  in  these  types  of  systems  must 
be  monitored  on  a  regular  basis  to  ensure  that  safety 
is  not  inadvertently  compromised  or  that  features  and 
functions  are  added  which  make  the  software  safety- 
critical 
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Projects  With  Mixed  Levels  off  Effort 


Not  all  projects  fall  into  neat  categories  for 
classification 

-  Some  projects  may  be  large  and  complex,  but  with  a  small 
portion  of  safety-critical,  low  risk  software 

-  Other  projects  may  be  small,  but  a  significant  portion  of 
software  is  safety-critical  and  high  risk 

Too  often  smaller  projects  argue  that  they  have  no 
safety-critical  software  out  of  concern  that  the  label  will 
lead  to  massive  amounts  of  work 

-  Partitioning  of  the  safety-critical  software  from  code  that  is 
not  safety-critical  allows  the  safety  effort  to  be  applied  to 
the  appropriate  safety-critical  portion 
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Summary 


Creating  a  safe  system  is  a  team  effort  and  safety  is 
everyone’s  responsibility 

Software  is  a  vital  part  of  most  systems 

Software,  by  itself  cannot  injure  you,  but  software  normally 
operates  in  an  electronic  system  and  often  controls  other 
hardware 

Software  is  hazardous  if  it  can  directly  lead  to  a  hazard  or 
is  used  to  control  a  hazard 

-  Hazardous  software  includes  all  software  that  is  a  hazard 
cause 

-  Is  a  hazard  control 

-  Provides  information  upon  which  safety-critical  decisions 
are  made 

-  Is  used  as  a  means  of  failure/  fault  detection 
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Tim  Kasse  -  Contact  Information 


Tim  Kasse 

QMS&A 

Program  Manager  -  Software  Quality  Assurance  Program 
National  Renewable  Energy  Laboratory  (NREL) 

1617  Cole  Blvd. 

Golden,  Colorado  80401 
(303)  275  -  3285 
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Lesson  Learned  from 
Pilot  Implementation  of 
Organizational 
Performance 
Management  (OPM) 
Process  Area 


Monday  Half-Day  Tutorial 


I  Specific  Goal  and  Practice  Summary 

SG  1  Manage  Business  Performance 

•  SP  1.1  Maintain  Business  Objectives 

•  SP  1 .2  Analyze  Process  Performance  Data 

•  SP  1 .3  Identify  Potential  Areas  for  Improvement 

SG  2  Select  Improvements 

•  SP  2.1  Elicit  Suggested  Improvements 

•  SP  2.2  Analyze  Suggested  Improvements 

•  SP  2.3  Validate  Improvements 

•  SP  2.4  Select  and  Implement  Improvements  for  Deployment 

SG  3  Deploy  Improvements 

•  SP3.1  Plan  the  Deployment 

•  SP3.2  Manage  the  Deployment 

•  SP  3.3  Evaluate  Improvement  Effects 


Organizational  Performance  Management  (OPM) 
purpose  is  to  proactively  manage  the  organization’s 
performance  to  meet  its  business  objectives.  In 
Maturity  Level  5  the  organization  is  expected  to 
continually  improve  its  processes  (methods)  and 
deliverables  (systems,  software,  services  and 
acquisitions)  based  on  a  quantitative  understanding 
of  its  business  objectives  and  performance  needs. 


Can  we  justify  a  tool  with  OPM 
How  to  do  it 


Some  Questions 

(from  second  case  study) 

|  We  have  only  financial  business  objectives 

f.  Our  PPMs  dose  not  provide  PMs  practical 
insights 

We  are  lacking  a  system  level  insights 

We  don’t  know  how  to  map  and  connect 
our  objectives  to  quantitative  targets  at 

process  /  deliverable  level 


m 


Case  Study  #1 
Requirements  Analysis 
and  Management 


Process  ROI 


roject  Idea  and  Proposal  Preposition 
Development 

If  an  average  developer  day  cost  is  -7000 
The  total  Program  effort  was  10220  day  (100%) 

The  testing  phase  was  1480  day  (14.5%) 

Defect  that  are  the  result  of  documentation  are  69%  of  all  defects 

If  we  will  assume  the  to  correct  69%  of  all  defects  will  take  around  40%  of  th< 
testing  duration;  ^  means  that: 

•  that  will  be  740  day 

•  With  the  overall  cost  of  5 1 8000 

However  to  add  100  review  days  in  the  static  tests  and  another  20  of  code 
inspection  will  end  with  the  cost  of  2100000 


And  still  we  have  saved  at  least  3080000  (440  days) 

Means  that  we  ware  able  to  reduce  4.5%  of  the  project  time 


Definition  of  Process 


rocess  Control 


Process  Levels 


Architected  and 


Objectives 

Structured 

Monitored  /  Measured 


Effective  /  Efficient 

Process  Interfaces  and 
Integration  in 
Lifecycle 

Prioritize  and  Balance 
Resource  Utilization 
within  Larger  Context 


and  Dimensions 

Improved  Process 


Suggested  Measures 

Architected  and  Improved  Process 


£• ;  Process  productivity 

||  Process  resources 

utilization  effectiveness 

|p  Process  resources 
'  utilization  efficiency 

Meeting  the  process 


objectives 

Other  processes  interfaces 
efficiency 

Process  related  defects 
density 


Operationally  Optimized  Process 


Process  Levels  and  Dimensions 


Known  Capability  and  Stable 

Defined  Ingredients 

Known  Critical  Elements 

Meeting  Objectives 

Controlled  Interfaces 

Responsive  /  Modifiable 

Resilience  /  “Agile” 

Relevant  ‘What  If  s  Scenarios 

Accepted  Tolerance  /  Freedom 
Boundaries 


Predictable  Outcomes 


Suggested  Measures 

Operationally  Optimized  Process 


Influence  of  Critical  Elements 
on  process  output 

Process  resources  utilization 
‘What  If  s  Scenarios 

Process  elements  capability 

Quantitative  definition  of 
process  ingredients 


Key  Questions 

•  What  is  the  current  performance? 

•  Is  this  value  ’’good”? 

•  Is  it  changing? 

•  How  can  I  make  the  value  “better”? 

Candidate  Attributes 

•  Definition  (completeness,  compatibility) 

•  Usage  (compliance,  consistency) 

•  Stability  (repeatability,  variability) 

•  Effectiveness  (capability) 

•  Efficiency  (productivity,  affordability) 

•  Predictive  Ability  (accuracy,  effects  of  tailoring  and 
improvements) 


Some  Examples 


Goal 


Measure 


The  number  of  process  elements  added,  changed,  and  deleted  during 
tailoring. 

Number  of  discrepancy  reports  generated  by  Quality  Assurance  audits 

The  number  of  process  elements  changed  within  a  specified  time 
interval. 

Product  quality 

Defect  leakage  to  subsequent  phases 
Productivity  (or  production  coefficient) 

Rework  as  a  fraction  of  total  effort 

Probability  distribution  for  an  estimated  quantity  or  related  population 
statistics 


Completeness 

Compliance 

Stability 

(volatility) 

Effectiveness 

Effectiveness 

Efficiency 

Efficiency 

Predictability 


elationships 
for  CoQ 


rocess  Quality  Audits  and  Progress 
Check  Calibration 

•  As  for  today  the  organization  is  managing  its  programs  as  large  and 
complex  programs  and  need  to  comply  with  more  than  just  one  quality 
standards  in  many  disciplines  (e.g.  HW,  optics,  software)  and  use  large 
groups  of  internal  and  external  assessors  that  perform  implementation 
checks,  progress  checks,  readiness  reviews  and  formal  appraisals. 

•  These  assessment  teams  are  typically  composed  from  groups  of  very 
experienced  and  professional  individuals  that  have  the  best  knowledge  in 
their  professional  domain  but  not  necessarily  on  how  to  conduct  an 
efficient  and  effective  appraisal  which  provide  meaningful  results 

•  The  combination  of  the  effort  and  expected  resources  increase  the  risks  on 
qualification  of  auditors,  domain  knowledge,  and  calibration  of  results  and 
findings  effectiveness 


ain  Steps  for  Process  Improvement 


During  our  analysis  and  planning,  we  were  able  to  identify 
improvement  targets  in  main  lifecycle  areas  such  as 

•  operations, 

•  information, 

•  governance, 

•  people 

•  organizational  structure, 

•  portfolios, 

•  project  execution, 

•  finance. 

And  as  in  core  process  that  are  critical  to  the  system  success  such  as 
stakeholder  management,  technical  interfaces  and  integration. 


rganizational  Background  and  Process  ROI 

1 

Quality  Audits  and  Progress  Check 
Calibration 

By  measuring  the  following  attributes,  we  were  able  to  increase  usability 
of  the  process  and  progress  checks  by  47%,  and  quality  of  deliverables  by 
37% 

Role  based  profile  and  criteria 
Calibration  mechanism  and  criteria 
Evaluation  mechanism  and  criteria 
Leveling  the  different  quality  engineers  and  ‘auditors’ 

Flowing  specific  trainings  (on  different  levels)  as  personal  development  and 
qualification  criteria 

Listing  specific  performances  as  indicators  for  leveling  justifications 

Structuring  the  different  audits  and  reporting  guidelines  in  a  single  mandatory 
to  follow  process, 


Process 

froject  Idea  and  Proposal  Preposition  Development 


The  Need 
Statement 


Project 

Feasibility  Check 


Project  Office 
Preposition 


Project  Team 


System 

Development  and 
Acquisition 


System 

Validation 
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sample  Projects 


%  From  ORG 


Sample  Practices 


£  %  From  Sample 


%  of  is  0 


%  of  >4 


%  of  <4 


%  of  is  4 


%  of  >6 


mean 


median 


mode 


Center 


19629 


#NUM! 


max 


sample  Projects 


Sample  Practices 


%  From  Sample 


mean 


median 


mode 


A1 


0% 

1 00% 
50% 
22 

21.15% 

3733 

19.02% 

526 

14.09% 

1575 

42.19% 

1626 

43.56% 

532 

14.25% 

779 

20.87% 

#NUM! 

4 

2 

7.058 


A2 


0% 

1 00% 
50% 

6 

5.77% 

957 

4.88% 

127 

13.27% 

476 

49.74% 

347 

36.26% 

134 

14.00% 

211 

22.05% 

#NUM! 

4 

6 

6.898 


Areas 


A3 

A4 

A5 

A6 

A7 

0% 

0% 

0% 

0% 

0% 

1 00% 

1 00% 

1 00% 

1 00% 

1 00% 

37.5% 

62.5% 

50% 

50% 

75% 

3 

13 

23 

13 

24 

2.88% 

12.50% 

22.12% 

12.50% 

23.08% 

647 

2069 

4961 

2914 

4348 

3.30% 

10.54% 

25.27% 

14.85% 

22.15% 

154 

195 

914 

378 

355 

23.80% 

9.42% 

18.42% 

12.97% 

8.16% 

213 

1092 

1850 

1413 

2528 

32.92% 

52.78% 

37.29% 

48.49% 

58.14% 

322 

705 

2358 

1165 

1305 

49.77% 

34.07% 

47.53% 

39.98% 

30.01% 

112 

272 

753 

336 

515 

17.31% 

13.15% 

15.18% 

11.53% 

11.84% 

82 

579 

775 

733 

1659 

12.67% 

27.98% 

15.62% 

25.15% 

38.16% 
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#NUM! 

#NUM! 

#NUM! 
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4 

4 

6 

0 

6 

0 
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8 

6.750 
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7.142 
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Interface  - 
SPEC  retest 
1% 

interfaces 

3% 


Defects  by  originator 


L  Zvn 


Requirements 
1% 


Cancelled  Defect 

796  Defect 

696 


Design 

2496 


Code  retest  &  Can  cel  eld 

1%  396 


aged  Process  for  Innovation 


Capture  Idea 


Enterprise 

Search 


Run  Portfolio 

Analysis 


Approval 


Build  Project  Team 


Execute  Project 

Design 

Operational  Potential 
Legal  Evaluation 


Feasibility  Case 

Strategic  Impact 
System  Potential 
Financials 
SWOT 


Customer  Feedback 


Finalize  Design 
Document 


Publish 

Operational  Case 


Community  Ratings  and  Reviews 


Deliver 


ur,  dr&jjntHMtb  ~ 


-a- 

aa 

pMBLIwfMU 

L  J 

r  ..-■Ax, 

Khpyl  >|M 


a 

IJittBTbhn 


ase  Study 


Fields 


Simulation 


Data  slot: 


m  AIIDataslots 
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Data  slots 
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Name 

Type 

Label 

Editable 

Required 

bd 

HDResolution 

Boolean 

H  d  resolution 

□ 

ScheduledDate 

Date 

Scheduled  date 

v 

m 

Attachments 

Document 

Attachments 

v 

a 

CustomerConta... 

String 

Customer  conta... 

a 

CustomerName 

String 

Customer  name 
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a 

Installation 

String 

Installation 

a 

Skid 
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TicketPriority 

String 

Ticket  priority 

Randomize  duration  using:  Normal  Distribution  (StDev^(none)) 
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(default)  Q 


2  hours 
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Value 


Unit 


Cost  per  unit 


Threshold 


Modify... 


Reset 


OK 


Cancel 


Help 
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Scenario: 
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Distribution  of  Probability 


Tine 


Normal 

* 

Constant 

Exponential 

Normal 

The  Normal  Distribution  should  be  used  when  observations  tend  to 
accumulate  around  a  particularvalue  ratherthan  spread  evenly  across  a 
range  of  values 


OK 


Cancel 
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Modify. 


Reset 


Name: 


OpenTicket 


General 
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ase  Study 


Simulation 


Scenario: 
Work  Time: 
Randomize  d 
Resources 


Name 


Distribution  of  Probability 


Exponential 


The  Exponential  distribution  should  be  used  when  the  probability 
of  observations  decreases  in  time 


Type 
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Cancel 


Process  ROI 


roject  Idea  and  Proposal  Preposition 
Development 

If  an  average  developer  day  cost  is  -7000 
The  total  Program  effort  was  10220  day  (100%) 

The  testing  phase  was  1480  day  (14.5%) 

Defect  that  are  the  result  of  documentation  are  69%  of  all  defects 

If  we  will  assume  the  to  correct  69%  of  all  defects  will  take  around  40%  of  th< 
testing  duration;  ^  means  that: 

•  that  will  be  740  day 

•  With  the  overall  cost  of  5 1 8000 

However  to  add  100  review  days  in  the  static  tests  and  another  20  of  code 
inspection  will  end  with  the  cost  of  2100000 


And  still  we  have  saved  at  least  3080000  (440  days) 

Means  that  we  ware  able  to  reduce  4.5%  of  the  project  time 


Process 

Quality  Audits  and  Progress  Check  Calibration 
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Notes 
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Case  Study  #1 
Business  Objectives 
Development  and 
Quantitative  Targets 
Management 


The  Challenge 


We  have  vision  and  business  strategy 

We  are  focused  on  financial  results  that  we 
can  report  on  P&L 

Why  we  need  other  ‘business’  quantitative 
objectives  and  targets 

What  are  the  benefits  for  the  system  and 
program  mangers 

How  we  do  it 


The  Concept  in  Plain  English 


aSPST*** -.A  .v-j.  ?,■ 


‘Cheshire  Puss,’  she  began,  ...  'Would  you  tell 
me,  please,  which  way  I  ought  to  go  from  here?’ 
That  depends  a  good  deal  on  where  you  want 
to  get  to,'  said  the  Cat. 


’I  don't  much  care  where  -’  said  Alice. 

’Then  it  doesn't  matter  which  way  you  go,'  said  the 
Cat. 

so  long  as  I  get  somewhere ,'  Alice  added  as  an 
explanation. 

‘Oh,  you're  sure  to  do  that,'  said  the  Cat,  ‘if  you  only 


waiK  long  enougn 


Tell  me  where  you  want  to  be  and  I  will  show  (measure)  you  the  way 


“When  people  consult  me,  it’s  not  that  I  am 
reading  the  future;  I  am  guessing  at  the 
future.  . . .  How  do  I  guess  at  the  future? 
Based  on  the  omens  of  the  present.  The 
secret  is  here  in  the  present.  If  you  pay 
attention  to  the  present,  you  can  improve 
upon  it.  And,  if  you  improve  on  the  present, 
what  comes  later  will  also  be  better” 


The  Alchemist  -  Paulo  Coelho 


The  Solution  We  Chose 


Strategic  Policy 
Deployment 
(SPD) 


Strategic  Policy  Deployment 


Combination  of: 


Clear  &  Aligned  Priorities 
Behavior  Changes 
Change  in  Thinking  (PDCA) 
Elimination  of  Waste 


. .  .to  achieve  Business  Results 


Snap  Look  to  The  Tool 


Strategic  Policy  Deployment 


A  process  to  focus  upon  Goals,  that  cut  across  the 
corporation 

Aligns  &  links  resources  &  action  in  pursuit  of  those 
Goals. 

Enables  progress  towards  the  Goals  to  be  measured 

Enables  rapid  root  cause  corrective  action  if  results 
vary  from  goals 

Drives  process  improvement 

Individuals  &  teams  get  clarity  on  their  impact  upon 
the  Goals 

It  becomes  the  yearly  implementation  of  our  long  term 
strategic  planning  process. 


Policy  Deployment  as  a  Tool 


Deployment  is  an  effective  tool  to  use  for  answering  the  following 
questions: 

•  How  do  we  identify  our  critical  goals? 

•  How  do  we  develop  plans  and  align  our  activities? 

•  How  do  we  communicate  our  goals  and  activities  level  by  level? 

•  How  do  we  align  the  abundant  talent  of  our  team  members  on  the 
critical  few? 

•  How  do  we  sustain  our  activities? 

•  How  do  we  quickly  change  course  when  required? 

•  How  do  we  learn  from  our  experience? 


Magnitude  of  Change 

Behavior  Change 

•  Discipline 

•  Emphasis  on  how  the  organization  will  deliver  the 
priorities 

•  Catchball  to  understand  the  priorities  and  the  means  to 
deliver  them 

•  Gemba  -  look  for  evidence  the  plan  is  proceeding  and 
in  control 

Clear  and  Aligned  Priorities 

•  Start  with  top  management  priorities  and  link/translate 
at  every  level 

•  Critical  few  metrics  match  Excel  commitments 

•  Must  deselect 


Your 
Start  Pc 

(Presen 


mfr. 


Step  1 

Choose  the  Focus 


Check 


Step  2 

Align  the  Organization 


Step  3 

Implement  the  Plan 


Operational 

Focus 


Stefjp^cfJiently 

Review  and  Improve 


ORGANIZATION 


YEAR 


irget  Priority  2 


Problem  strip 
for  red  items 


The  Forms  Link. . . 


merit  Sheet 


Red 

items 


Program  Plan  (multiple) 


Tracking  Sheet 


jr  Process  to  build  consensus  through  dialog  about  the 
|  f  goals  and  how  to  achieve  them. 

| !  Two  way  communication  that  arrives  at  a  collective 
wisdom  on  the  priorities  and  the  plans  to  deliver  the 
Bi  results. 

1  Leader  needs  to  have  a  vision  of  what  is  needed  and 
how  it  may  be  achieved. 

•  Team  will  provide  input  on  the  specific  how. 

A  •  The  leader  will  confirm  the  plan: 

Push  the  team  to  stretch  further  if  the  plan  comes  short  of 
what  he  had  in  mind. 

/]  Question  and  develop  understanding  of  the  plan  if  the  plan 

jCK  exceeds  what  he  had  in  mind. 


T  argets 


II  priorities  require  a  target  so  they  can 
be  measured. 


Targets  have  to  be  achievable,  challenging, 
based  on  reliable  data,  and  SMAR" . 


S  -  specific 


R  -  realistic 


-  timed 


M  -  measurable 
A  -  agreed 


The  DO  Phase 


Check 


Review  the  plan  on  a  regular  basis. 

•  Look  at  metrics  daily/weekly 

•  Formal  reviews  monthly 

results  vs.  expected,  as  well  as  the  countermeasure  to 
fill  the  gap 

Ask  why  if  the  team  is  doing  things  that  are  not  in  the 
plan. 

Question  frequently  by  going  to  see. 

•  Schedule  the  time  to  look  for  evidence  that  the  plan 
proceeding  and  in  control. 


w*m 


,  The  CHECK  Phase 

?■ 

[g  reviews  maintains  the  discipline  of  the 


Chec 


A  mini  PDCA  cycle  takes  place  everyday  as  activities 
are  checked  constantly 


Counter  measures  are  data  driven  looking  at 
root  cause. 

Check  if  previously  identified  counter 
measures  are  working  and  on  track. 

Don’t  react  to  noise. 

Escalate  issues  that  can  not  be  resolved  to 
the  next  level. 


E 


The  CHECK  Phase 


The  CHECK  Meeting 


manager  runs  the  review  meeting. 


The  manager  should  focus  on  standard  work: 
•Metrics  Chart 
•Action  Plan 

•Corrective  Action  for  RED  Items 
•Key  Items  at  Risk 


•  Asking  clarifying  questions  during  the  review 
process. 

•  Making  sure  that  each  person  knows  what  is 
expected  of  them  to  move  forward. 

•  Conf  irmation  check  -  will  this  plan  get  us 
there? 

•  Follow  up  at  Gemba  before  the  next  review  for 
key  issues. 


Check  Questions 


Policy  Deployment  -  Questions  to  Ask 
Do  you  have  a  plan? 

Does  the  plan  close  the  gaps  to  the  goal? 

Is  the  plan  being  executed  on  time? 

Is  the  plan  generating  the  expected  business  results? 


£>  If  the  answer  is  no  for  any  of  these,  generate  a  countermeasure 


Monitor  effectiveness  of  countermeasure 


Look  for  trends  -  not  just 

red/green 


•  Metrics  chart  gives  an  overall  view 
if  on  track 

•  Use  graphs  and  charts  to  see 
what  is  really  happening. 


The  ACT  Phase 


ft  off  track: 

•  Review  the  plan  and  countermeasures  to  confirm  that  gap 
from  the  plan  can  be  closed. 

Is  the  plan  itself  valid  or  does  it  need  to  be  modifid? 

?  •  Follow  up  to  determine  if  actions  on  the  countermeasures 
have  been  done. 

•  Go  to  GEMBA  to  check  to  see  if  progress  is  being  made 
on  critical  issues 

If  on  track 

•  Lock  in  the  condition  with  standardized  work 
Confirmation  Check 

•  Does  the  plan  and  countermeasures  link  to  the 
goal/vision? 

•  Does  it  seem  reasonable? 


Check 


How  We  Did  It 


<BU  ABO  Business  Objectives 


The  strategy  document  has  53  quotes  the 
leading  us  to  an  optional  list  of  Business 
Objectives  and  Quantitative  Targets 


:  Supporting  professional  and  personal  service  on 
competitive  terms 

Flexible  adjustments  to  changing  market 
conditions  at  the  lowest  possible  cost  and  a 
satisfactory  time-to-market 

Keep  its  leading  position 

Group’s  systems  must  be  capable  of  handling 


growth  in  the  Group’s  business,  organically  as 
well  as  through  mergers  and  acquisitions 


Business  Objectives 
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Business  Objectives 


Prioritizes  resources  and  projects  based  on  cost- 
benefit  considerations 


Manages  the  actual  development  process 

h 

Systems  must  be  designed  for  group-wide 
deployment 

Systems  and  functionality  are  reused  across 
products,  distribution  channels,  brands  and 
markets 


Systems  must  optimize  cross-organisational 
processes  and  make  it  possible  to  combine  parts  of 

i 

the  Group’s  products  into  new  products 


Business  Objectives 


By  reusing  system  elements,  even  across  different 
technical  platforms,  significant  efficiencies  are 
gained  in  the  development  of  systems 

Integrate  third  party  systems  into  the  whole 
system  complex,  regardless  of  the  technical 
deployment  platform 

Minimize  the  costs  associated  with  the  integration 
of  applications  and  tools  across  systems  and 
platforms 

Limited  but  adequate  set  of  market  leading 
technologies  are  used  as  standard  tools 


Business  Objectives 


SyDLC  and  TCO  must  be  implemented  in  such  a 
way  that  the  integrity  of  the  business  cannot  be 
compromised 

level  of  security  and  operations  must  be  high  and 
financially  sound 

Systems  and  platforms  must  have  a  high  quality 
level,  protecting  the  Group  against  errors,  down 
time,  security  breaches  and  data  loss 

Quality  level  must  correspond  with  risks, 
consequences  and  not  least  the  expectations  of  the 
customers 


Business  Objectives 

Systems  must  adhere  to  the  agreed  service  levels 
and  be  delivered  with  the  agreed  functionality 

Simple  and  accessible  user  interfaces,  adapted  to 
the  user’s  role  or  the  customer’s  needs 

Access  is  given  to  the  necessary  functionality  and 
information  from  the  underlying  business  system 
based  on  consolidated  data 

Systems  must  constantly  support  the  chosen  set  of 
distribution  channels  and  user  interfaces,  enabling 
the  Group  to  meet  the  customer  at  any  given  point 


Business  Objectives 


Knowledge  about  the  customers  and  their  behavior 
must  be  gathered  in  a  structured  way  on  each 
customer  interaction,  and  related  to  the  Group’s 
products 

Integrated  and  customer-facing  sales  and  advisory 
system  ensures  that  products  and  services  can  be 
developed  and  deployed  across  business  units, 
customer  segments  and  distribution  channels 

Reduce  the  Group’s  costs  by  optimizing  the  whole 
value  chain 


Costs  associated  with  the  rationalisation  of  processes  must 
be  minimal,  enabling  economically  feasible  automation  of 
even  small  business  processes 

Business  procedures  must  be  implemented  direct  as 
supported  processes,  guiding  employees  and  customers 
through  the  activities  with  as  little  prior  knowledge  as 
possible,  letting  them  concentrate  on  the  products  and 
actual  business. 

Enables  conversion  of  manual  activities  into  automatic 
sequences  without  changing  the  basic  design  of  the 
underlying  processes. 


Business  Objectives 


Business  Objectives 


Combine  activities  efficiently  and  flexibly  across 
distribution  channels,  partners,  brands  and  markets, 
wherever  this  is  desirable  from  a  business  point  of  view 

Systems  must  support  the  processes  which  gather, 
organize,  share  and  analyse  the  entire  knowledge  platform 
that  exists  about  customers,  products,  business  initiatives, 
organization,  employees,  etc 

Information  must  be  available  at  any  time  and  anywhere  to 
those  it  is  meant  for 

Group’s  management  processes  and  pricing,  they  must  be 
based  on  consolidated  and  sufficiently  current  data 


Business  Objectives 


Increase  flexibility  gradually  without  compromising  on 
efficiency  and  stability 

Diversity  is  handled  systematically  and  efficiently  by  using 
an  infrastructure,  which  efficiently  integrates  systems, 
processes  and  manual  activities  across  platforms  and 
technologies 

Infrastructure  is  provided  to  developers,  freeing  them  from 
having  to  programe  integration  and  flexibility  into  each 
system 

Use  of  market  leading  standards 

Design  of  system  elements  focusing  on  flexibility 


Business  Objectives 


System  elements  must  be  designed  to  scale  in  line  with 
business  growth  and  expansion 

System  elements  must  be  capable  of  handling  unexpected 
events 

Ensure  that  systems  can  continue  normal  operations  with 
the  least  impact  on  the  business 

Business  continuity  during  normal  operating  conditions  as 
well  as  in  disaster-like  situations 

Systems  design  must  if  possible  take  into  account  the 
changeability  of  externally  controlled  data  and  processes 

Readiness  for  change  by  implementing  changes  for  the 
entire  group 


Business  Objectives 


Resources  can  thus  be  reused  in  any  other  project  or  area 
in  the  Group  in  a  simple  and  efficient  way,  thereby 
ensuring  consolidation  of  both  data  and  functionality 

It  must  be  possible  to  combine  scattered  IT  resources  into 
complete  systems,  applications  and  actual  business 
processes 

Infrastructure  must  handle  the  coupling  dynamically  and 
parameterized 

Selection  of  coupling  method  must  not  be  based  on  a 
technology  choice  made  by  the  developers 


Business  Objectives 


Infrastructure  and  development  methods  must  as  a 
minimum  support  a  layering  of  systems  into  user 
interfaces,  business  logic  and  data 

Service  levels  must  if  possible  be  based  on  dynamic  and 
flexible  policies,  which  are  directly  definable  in  the 
operational  environment 

Infrastructure  must  efficiently  handle  error  detection  and 
quality  control  of  complete  system 

Infrastructure  must  efficiently  support  the  integration 

Architecture  is  an  essential  parameter  when  choosing  a 
third-party  system 


Business  Objectives 


Program  managment  must  ensure  consistency  across 
manual  and  system-supported  processes,  by  enabling  any 
given  process  to  involve  both  manual  and  automated  work 
items 

Infrastructure  must  provide  simple  and  efficient  methods 
for  supporting  business  procedures,  processes  and  routines 
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Kobi .  Vider(a)hotmail.  com 


Phone:  +972522946676 


Leveraging  Your 
Service  Quality  Using 
ITIL  V3,  ISO  20000  and 

CMMI-SVC 


Monday  Half-Day  Tutorial 


Service  -  Employment  in  duties  or  work  for 
another 


The  Challenge 


This  situation  where  organization  is  running  a 
system  lifecycle  a  matrix  with  internal  or  external 
contractors  =  service  providers,  with 

•  separate  quality  management  systems  and  with 
compliance  to  different  standards  (e.g.  AS9100c)  and 
qualification  (e.g.  MIL-STD  217)  on  different  parts  of 
the  system  /  product  lifecycle 


The  Challenge 


This  situation  where  organization  is  running  a 
system  lifecycle  a  matrix  with  internal  or  external 
contractors,  with 

•  With  partial  overall  view  in  interactions  and 
handshakes  between  these  groups  is  introducing 
inefficient  usage  of 

resources, 

expensive  maintenance  of  duplicate  infrastructures 

and  Organizational  Sets  of  Standards  Processes  as  well  as 

assets, 

•  May  result  in  less  quality  and  impacting  the  end 
product  /  system. 
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The  Lifecycle  Challenge 
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The  Theory  in  the  Models  is  Nice 

However 

Real  Life  is  More  Complicated 

Much  More 


Common  Failures  -  1 


ganizational  risk  events  are  predominantly 
managerial,  not  technical. 

Lack  of  defining  business  objectives  in  quantitative  terms  and 
structure 

Inadequate  definition  of  ’Good  Enough’  level 

Inability  to  differentiate  different  business  objectives  and 
success  factors  for  the  different  domains  and  lifecycle  phases 

Inadequate  resource  usage  and  adjustment  to  Plan  and 
Objectives 

Failure  to  identify  and  manage  risks 
Poor  or  mismanaged  service  /  operational  requirements 
Uncontrolled  baselines,  no  configuration  management 
Misunderstood  business  /  operational  needs  and  objectives 


Common  Failures  -  2 


Poor  contractor  acquisition  or  management 
Lack  of  skills,  capability  and  training 

Poor  planning  and  tracking 

•  Value  Stream 

•  Equipment 

•  Resources 

•  Finance 

Poor  /  misuse  of  data  and  measurements 
Inability  to  estimate  accurately 
No  quality  assurance  /  control 
Poor  communications 


The  Operational  Need 


Management  capability  level  from  both  professional  and 
knowledge  level 

Performance  and  reporting  norms 

Self  management  and  self  discipline  maintaining  personal 
professional  and  knowledge  capabilities 

Individual  and  team  discipline 

Cooperation  and  knowledge  and  resource  sharing 

Appropriate  visibility  of  information,  data  and  capabilities 

Quality  of  readiness  and  preparedness  for  performing 
mission 


The  Approach  to  the  Solution 

Concept 

Best  practices  in  the  model  focus  on  activities  for 
providing  quality  services  to  the  customer  and  end 
users 


To  identify  improvement  targets  in  main  lifecycle 
areas  such  as  operations,  information,  governance, 
people  and  organizational  structure,  portfolios, 
project  execution,  and  finance 

Select  processes  that  are  critical  to  the  system 
success  such  as  stakeholder  management, 
technical  interfaces  and  integration 


The  Approach  to  the  Solution 

Concept 

Build  an  action  plan  composed  from  the  following  main 
steps 

•  Organizational  map 

•  Functional  team  and  groups  size  and  role  in  the  lifecycle 

•  Full  lifecycle  map 

•  Setting  improvement  targets 

•  Gap  analysis 

Suggesting  to  the  senior  management  to  address  the 
lifecycle  and  process  (as  a  whole)  as  a  complex  of  crossing 
interfaces  and  to  add  additional  content  to  the  lifecycle 
map  (as  a  layer) 


The  Conceptual  Solution 


Building  on  contingency  theory,  it  outlines  a 
comprehensive  framework  suggesting  a  fit 
between  the  level  of  Mission  interoperability  and 
environmental  as  well  as  internal  contingencies. 

Moving  from  the  current  environment  of  basic 
process  and  way  of  thinking  toward  a  more 
controlled  and  measured  process  to  reduce  the 
overwhelming  amount  of  information  that  build 
decisions 


JThe  Proposed  Solution  Concept 

Using  the  CMMI-SVC  as  an  overall 
umbrella,  to: 

•  Increase  results  and  effectiveness 

•  Reduce  quality  related  activities  costs  by 
reducing  overlaps  and  choosing  the  appropriate 
parts  only  as  part  of  the  ‘whole’ 

•  Reduce  administration  costs  by  improving  the 
ability  to  manage  the  lifecycle  network 

•  Converged  working  network  helps  businesses 
to  save  procurement  costs  of  infrastructure 


Process  Improvement 
Effort  Objectives 

Group  Target  is  Process  Improvement: 
Increase  Processes  Efficiency 
Increase  Budget  utilization 
Reduce  Cost  of  Poor  Quality 
Increase  Uniformity  in  Processes 
Leading  Standards  to  Compliance  with 
ITIL 

ISO  20000 
ISO  25999 


Mapping  Sample 


IT  Infrastructure  Library  -  ITIL 


Is  “best  practice”  in  IT  Service  Management, 
developed  by  OGC  and  supported  by  publications, 
qualifications  and  an  international  user  group 

Assist  organizations  to  develop  a  framework  for  IT 
Service  Management  and  to  certify  the  service 
managers 

Worldwide,  most  widely  used  best  practice  for  IT 
Service  Management 

Consists  of  a  series  of  Core  books  giving  guidance 
on  the  provision  of  quality  IT  services 
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ITIL  Processes  &  Function 


ITIL  Processes 


Service  Support 


Service  Delivery 


Service  Level  Management 


Availability  Management 


Capacity  Management 


IT  Service  Continuity  Management 


Financial  Management  for  IT 
Services 


ITIL  Functions 


Service  Desk 


Incident  Management 


Problem  Management 


Change  Management 


Release  Management 


Configuration  Management 


What  is  ISO  20000 


ISO  20000  can  be  summarised  as: 


standard  to  promote  the  adoption  of  an 
integrated  process  approach  for  the  effective 
delivery  of  managed  services  to  meet  business 
and  customer  requirements 


set  of  “controls”  against  which  an  organization  can 
be  assessed  for  effective  IT  Service 
Management  processes 


The  ISO  20000  standard  defines  the  requirements 
for  an  organization  to  deliver  managed  services 
of  an  acceptable  quality  for  its  customers 


Structure  of  ISO  20000 


he  Standard  is  divided  into  two  distinct  parts: 


Part  1  -  provides  the  requirements  for  IT  service 
management  to  gain  certification 


Part  2  -  Code  of  Practice  for  Service  Management 

•  Provides  guidance  to  internal  auditors  and  assists  service 
providers  planning  service  improvements  or  preparing 
for  audits  against  ISO  20000 


ISO  20000  Processes 


Management  Systems 


Management  Responsibility,  Documentation 
Requirements,  Competences,  Awareness  &  Training 


Planning  &  Implementation 


Plan,  Implement,  Monitor,  Improve 
(Plan....  Do....  Check . Act . ) 


Planning  New  Services 


Planning  &  Implementing  New  or  Changed  Services 


Capacity  Management 
Service  Continuity  & 
Availability  Management 


Service  Delivery  Processes 

Service  Level  Management 
Service  Reporting 


Release  Processes 


Release  Management 


Control  Processes 

Configuration  Management 
Change  Management 


Information  Security 
Management 

Budgeting  &  Accounting  for 
IT  Services 


Resolution  Processes 

Incident  Management 
Problem  Management 


Relationship  Processes 

Business  Relationship 
Management 
Supplier  Management 


ITIL  Service  Support  Processes  &  Functions 


ITIL 


Service  Desk 


No  formal  Process 


ISO  20000 


Resolution  Processes 


Incident  Management 


Problem  Management 


Change  Management 


Configuration  Management 


Release  Management 


Control  Processes 


Release  Process 


ITIL  Service  Delivery  Processes 


ISO  20000 


ITIL 


Service  Level  Management 


Service  Reporting 


Service  Level  Management 


Business  Relationship  Management 


Supplier  Management 


Service  Continuity  &  Availability 
Management 


IT  Service  Continuity  Management 


Availability  Management 


Budgeting  &  Accounting  for  IT 
Services 


Financial  Management  for  IT 
Services 


Capacity  Management 


Capacity  Management 


Information  Security  Management 


No  formal  Process 


CMMI  Calibration 
Process 


[M 


T  U  V  VV  X  T 


I  M  M  AC 


CM  Ml -SVC  V  1.2 


3  ’";:□]  Vb-  uj.sj  Ard  CAT  Puif-c-se 

3  Pmcos  VBiCausd  And  CAW  1 

3  Treims  VB-Causd  Ard  CAT  1 


The  foifcse  tit  Cau  j a  Ard-ysis  ri  KapJudpn  (CAT)  ,j  it  j d >=n Cl I*y  ej 
b-£  Ci*::  j:'.  :  -  Ec  pi^wC^nn  fre-m  e-m.mr|  -  tcftiHiiE 

KCsmiri  Cl. 33  cJ  IOJnzz]  bl  ?mfcloni 

Kcc:mia  cf  Cbtel  bC  pmfclcrnj  in  syslnndjrdlv  ±^l3m(r*=i 
3d  -r:  >',Tj  m-Cl  Per  Warn 
1.1  Sd^  Cd'ert.  b-C  pretlmi  'e-r  j-j  yi  s 


1  fiscal  VBCKd  Ard  CAW 
3  ’recas  VBCausd  ArdCAT 


I.  Ca? ■=■  n! cvb :  i :J— "  a- £  prsfelem  diji 

J.  Jf-grairc  te  Cb1-^  b£  Fnfcl*3iu  ~j  t;B-dy:^  'urt-ig 


i-Cj.u  .ArdCAT 


: J  id  zz'zz  1  :Jzz  j  ■ 


-:  :- 


3  zmeas  VBCausd  Ard  CA’ 
3  J  re  ecu  WBrCMisd  ArdCAW 

3  ’em  WBiCBiid  Ard  CAW 


3  ’em  VnCKid  And  CAW 

3  PIxaiMCudArdCW 


1.2  1.  Conduct  EB-sd  ardysis  wit  tesc  rapenifclc  fo^msiri  te  Cok 
1.2  2.  Andyse  s  decree  £cJ cc  j  b£  pnfcicms  ?  tcinnirc  tarrec:  u.ia 
1.2  3.  Ce.:  id— Ccj  z eJ— j  iri  pritlerni  :n:’  cr  ?*:re:'  y.is 

A.  ’ec:i:ri  z  :  z  _  — zr~.  mz\  c  r  s  i  fee  Idea-  te  Fr=r=-"  te  iliCi-re  e=crr3-cc  c' siitiiIb  £d*^S  b£ 
1.2  pn^loru 

Address  Cmud  nf  OfcfccEi  ar£  Problems 

Teclca-sa  c'dd'edi  ard  pmfclcjnj  an  syslar  fj:dly  BiCrasaZ  Cc  prcYor:  tar  future  ceeLneree 
mpJaF:-':  A:1 :r  ’e:  :i j i 


3  ’em  VB-CBLid  ArdCAT 
3  ’em  Vb-  Cj.sj  And  CAW 
3  ’em  VBCausd  ArdCAT 

3  Jerai  V'B-CBLid  And  CAT 

3  ’em  VB-CB-id  Ard  CAT 


2.1  1.  Ard-e  able-  pretests  b-C  CCamirc  tor  pncn'jn 
2.1  2.  aJni  Blicr  prepesds  C=  te  implerr  erzi 
2.1  3.  C -i:  jeC e r  iConi  'cr  implcmcr  jr|  *c  ^Dcr  prepest: 

A.  finrC'vB-i  rerre-re  similB  ue'aj  b-£  pcblcci  tt:  may  e 

2.1  FreduECi 

2.~.  3.  Ca-cV  b-£  Ccn-rnCT’C  imFnTOng-C  Fnpcidi  fler  J*c  gi^B-uii'icr'i  iiJ  cJ  j Cb-£b£  p 


[  .  j".:  t  e  eJ  C.‘ i-]3 

 j” :  at  !:-j-yc  >;s 


»>.n  te  ef-B-|c  ir  Fe^rrmi1  cc  cJ  te  prelects  C  clrcd  precr 


u  Vb-  Cj.id  .Ard  CAT 

13  VB-Ci.id  Ard  CAT 


eras  Va-Causd  .Ard  CAT 


2.3  ’  rec-rd  ci.31  j-dysu  b-C  ppLciuticr  £aa  J:  •  . 


The  purpese  eJ  Cerliurtlcr  V  j-  i :■- T  (CJil)  is  J=  re -dl 1 1 1 h  b~l  mr'jr  te  i r ".ejn ty 
Feitfii  usiri  cerliurt"jer  ida-alniiicr,  ccrlp-rt-jer  ccricl.  r :  -  J  i  _  ■  si  =  -  sJMus  ■ 
2  1  -  :  :  : Ci-'j.ji  C=’  Furpc-sjc  ‘j".  :  -  i_  i  j 

LO  Harmonization  Like  Cl 6  Comm)y^~Ll  Harmonization  (Part  Coom) 


CMMI-ACQ  V  1.2 


Ccr  ViTiPrec  Fret  Fret  SQIT  3P  Hum  Tide 


ACC  3  S:f|Cb.:CA’ 

iACO  3  3up|  Cflu:  CAT 

ACC  3  S-piCb-iCAT 


AhDC  3  l-.pi  Cd.:  CAT  i 
ACC  3  l-F|Cd.:CAT  1 


The  F’.rppAC  eJ  Cj.i j  Ardyais  b:£  TjaeluacfT  (CAW)  13  lb  iiSEX-j-V  u.ia  cJ  lie'ccj  j 
e+er  pntlemi  b-£  Cdrc  j:*.  :  -  Cc  prcra-C  tan  V=m  ecD.mri  ir  lllc  '.CLrc 


a  cJ 

1  ncc:cB.ia  c '  fletel  B-C  Clhcr  prcfclcm  bc  lycacnaaedlv  CdmmrcC 
ld-m  S*bC  Vj  Ardyiu 
1.1  IdceC  j.  aril  ct1:-  :  ntlemi  Jer  B-dyiu 


1.1  l.  Ci*—  rd-.'B-Cid'ciCcrFnfcl’m  ill 

1.1  2.  Jifamir;  Jrc  b-£  eta*  Fnfclari  b  fccB-dyc^  '‘’■rf-g- 


.ACC  3  1.::  Ci. 


ACC  3  3up| 

.ACC  3  3Lp|i 

ACC  3  Slpi  C  j. 


CAT  ] 


Aflrfps  CBU3E3I 

P'a^enr  Qr.id  J "  1 1  : J  :  :  : : i  mri  ei-  =-  iretJari  rt  prepeie  Biers  b 

1.2  bCimteri 


CAT  1  1.2  1.  Crr£LC-  ed.id  B-dpii  wii  these  raperiitlc  !tr  pa'errruri  +c  "Bit 

CAT  i  1.2  2.  Ard-|CC  sd^li;  b-£  e  +  a*  prctlem]  i  CcCcimire  Jinr  me:  e».sa 

CAT  1  1.2  3.  Creep  3 deem  £eJ*=j  B-ii  c  +  cr  prctlem 3  tuei  :r  *arrce:=d.sa 

■1.  Frc-pcic  b-C  CcB.ma-:  j:-.  :  - 3  It  tc  J Tz  pera-:  ■Jrc  ^  j.re  em-rra-ce  e'  simila* 
ACC  3  5i FI  Cm;:  CAW  1  1.2  lic'jj  cr  c  J*cr  pn t lems 

ACCras  Causa  c'  Cc'zcj 

Tee"  ».!□  z J  u z'z: "j  btC  c^a*  pntloTis  an  SYS^emf-rcdly  B2£-aie£  —  piusu-'  J*nr 
ACC  3  3.piCa.:CAT  2  2  future  c ecu rra-cc 

fib-"  Abler  ’ e :  :’J : 


ACE  3  3up|  Ca_:  CAT  2 
ACC  3  5up|  Cd_:  CAT  2 
ACE  3  Sufi  Cau:  CAT  2 

ACC  3  3up|  Cd.:  CAT  2 

ACC  3  Sufi  Cau:  CAT  2 


2.1  1.  A- a.*.”  dr".  pnpesds  b-C  Ctbrnirctorpncnla 
2.1  2.  Selad  Bdjcr  prepcsd]  --  fee  implcm cried 
2.1  3.  C  ejz  del  -  e  Tars  Per  impJcmcrari  [he  Bjcr  pnpesds 
A.  £er  jJv  aru  rme*n  iimila*  uz'-as  7-a;  may  1=33:  ir  e  +  a-  pi 
2.1  preCudi 

3.  :£cr  j\  b-£  Cecumer:  impreraner:  pnFZsd:  Jer  It—.  :ia- 
2. 1  Fnpqja 


1  iC:Ji'a£it 


SyduAb  I"*:  eJ  C^aria 

Z  Pvd  u  j'r  +c  e-".m  eJ  :‘rja  cn  | 


=  F’^’T 


ACC  3  Sufi  Cm::  CAW  *2 
■ACC  3  S.p|Cau:C.AT  2 


1.  Venure  te  rha-p  ir  per’erm B-ce  cP  !tic  pn|ec^i  CcPircC  p 
2.2  uitpncaiai  «  -pFn=pn^c 

2.  Vani^rc  J-c  cjpjtiluv  e?  prciadfa  ^dfirsfi  pr=  saa  cr  cT  a 

2.2  HFpnspnifc 


ACC  3  5-PiCff-:CA^ 


TokhttS  ?j‘j 

2. a  *QKn£  GV.id  ard- fin  art  ra,d«-1cr  Jcr  uvz  ar. 


.VIC  2  S-p|  Ccr  CM 


■rffc  purpose  cJ  CcrljLnrtcr  ’ s-“  »'CVj  ia  Ic  alifcha'r  art  marOiR 
ir“jq|n?^  cJ  w*=r4c  preCi-rt  '-airj  ccr'ip-rajcr  il:q- j Idijct.  o:rJi||LraQ':r  ccr >=l 
ccr^p-r^cr  aX.a  Koc.r'jri,  art  OEr1f.nQcr  ■v.Aili 


CMMI-DEV  V  1.2 

Ccr  hM  Frei  Pm  Pm  SO  '  3?  Hur  Tide 

:s 

a  s-p  c*.  cv^ 

The  purpose  tit  Causd  Andysis  a-d  Taeluder  (CAT)  is  1:  identi 
pnhlemi  B-t  "Jre  Bier  "jc  pre-^z-T  tor  Jnm  ecc.mri  ir  life  ' 

:r 

S  i^p  Cm.  CAi 

Pec:ea.sa  zJ  uzJ— j  ari  p+o*  pnfclons  an  i -.3 ".en a : :ai *.'  u el 
l  1  A  me:  cause  is  a  icurec  ct  a  de'ccl  surfi  +a_.  if  iris  iniEnll,  dr 

"E  - 

a  s-p  o-  cm 

a*f.  :ij  Jrr  A-jrys  i 

3  1.3  Sda::  'he  £  i'll  arfi  tft  pretl'ms  fci  B-dyais. 

:: 

5  I-  :  Cau  C  =.^ 

1  1.  L  Ci?—  rdcwB-:  id'-^  er  pnfclem  ujli 

:e 

SipCAr  CA‘' 

1  1.1  nomine  v.+lich  defect  Bid  eAp  pmblons  will  he  B-dyscd  fui 

is 

a  5.P  Cj.  ■!=.’ 

.v-  C«.ia 

3  3.2  ?  .t-  dm  j  j“j  mfi  a  :J  ic  t  zJsz'mj  art  prcfclcm  a  a 

:e 

5  S_p  Col  C  =.* 

1  1.2  CerdLc:  ca.sd  i-ipis  wi?  ?cpeeplewhe  Be  rape-rsifeJe  fcrl 

:: 

3  S-p  C*.  t&* 

1  1.2  .A-d'iee  idecjd:  b-£  c  +  o-pretlons  b  CStmirctarre 

s  s_p  Cj.  ■:  =.* 

1  1.2  Qeup  ? e  i d  — "jzi  l: zj— j  b-£  c  +  ct  prefclors  zasrZ  cr  ?ejr  rci 

: : 

S  S-p  Cm.  C  =.’ 

Aiepesea-i:  desimml  aders  ?ilri^  e  fee  "aka-  "o  FrwrV 
1  1.2  eretCTFnfclrans 

:: 

S  S-p  Cm  CA* 

Audios  Ca.sa  cf  ?eJ-z:  j 

T:c':j.sp  z  j  ::jz  j  ar£  c?er  pretlcms  az  s  yz  xxn  a  j  ed  I  *,  bz  u 

2  2  cuLinra 

IS 

a  S-P  Cm  CA' 

te  Abler  ^repesds 

:: 

a  S-P  Cm.  CA* 

2  2.1  1.  Ardpetcblcr  pnpesds  b-C  Cebmire  tar  pncnla 

a  S-P  Cm-  CA' 

2  2.1  2.  Sdr=:  te  Biler  pnpesds  ti:  mil  fee  implorT-.^ 

:i 

a  S-p  Cm-  CAIf 

2  2.1  3.  ur-iaz  e'::r  izir  s  Jer  implsne-jri  te  Bier  pnpesds 

:: 

a  S-p  Cm-  CAP 

2  2.1  A.  derEify-id  mmcrc  simila*  dcfccLi  dT«C  may  izesC  ir  ether  pm 

:: 

a  S-p  Cm-  CA^ 

2  2.13.  Ca-'j'y  B-C  dpoimo-:  impievon a-:  pnpesds  Jer  te  er*n: 

.. 

a  S-P  Cm-  CA' 

hi*A  _  jl:  tz  ^  im:  e  J  'll"  a-  J3 

:e 

a  s-p  cm.  ■:  =. ’ 

2  2.2  1.  Voni-rc  >•=  c^m-ic  ir  >■=  par^crmm-cc  c-  J?c  prc|«3=?3  islra 

:e 

a  S-p  Cm.  CAP 

2  2.2  2.  Vbbuie  tf!c  cjcjr=  1  :>■  cf  *c  =-=.=:  a  iScfirnS  precaa  b  j:: 

is 

a  S-p  Cm-  CA* 

Iq-c i-4  IjCi 

2  2. a  '"'‘odc'.'S  :Cjr.aa  j- j  ‘fi  a.  ari  ras!1. ■  tad  Jcr  «-a-c  az—ii  "?*«=  pre 

:: 

2  S-p  Ccr  CM 

The  purpese  cJ  Cerlp_rtlcr  Ywr^ji.er'!  'c  '■')  is  b  a ~dt list  a 
pnCue-j  usir|  eerliurt-jcr  iCerlfica-jer.  z-: r ' ral e r  cerTrc 
arC  eer'iaurdler  a.Cij 

1  Harmongatiori  Other  PA  (Map)  X  CM  M  Is  G  P  s  CM  M 1-5  VC  VI .  2  X  CMNI-ACpyi.  2  /  OMMI-OEV  VI  .2 


First  Level  Filtering  (PA  Level) 


DEV 

ACQ 

SVC 

.  Project  Planning 
'  Project  Monitoring  and  Control 
:  Process  and  Product  Quality  Assurance 
Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

Project  Planning 

Project  Monitoring  and  Control 

Process  and  Product  Quality  Assurance 

Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

Project  Planning 

Project  Monitoring  and  Control 

Process  and  Product  Quality  Assurance 
Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

in 

Organizational  Process  Definition  +IPPD 
Organizational  Process  Focus 

Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management +IPPD 
;  Risk  Management 

Organizational  Process  Definition 

Organizational  Process  Focus 

Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Risk  Management 

Organizational  Process  Definition 
Organizational  Process  Focus 

Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Risk  Management 

‘  s 

Quantitative  Project  Management 
Organizational  Process  Performance 

Quantitative  Project  Management 

Organizational  Process  Performance 

Quantitative  Project  Management 
Organizational  Process  Performance 

Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Supplier  Agreement  Management 

Supplier  Agreement  Management 

Requirements  Development 

Validation 

Verification 

Acquisition  Requirements  Development 

Acquisition  Validation 

Acquisition  Verification 

Technical  Solution 
Product  Integration 


Solicitation  and  Supplier  Agreement  Development 
Agreement  Management 
Acquisition  Technical  Management 


Capacity  and  Availability  Management 
Incident  Resolution  and  Prevention 
Service  Continuity 
Service  Delivery 
Service  System  Development 
Service  System  Transition 
Strategic  Service  Management 
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P  DAR  RSKM 


inancial  Management 

for  /' 
IT  Services  // 


Capacity 

Management 


/  IT  Service 
Continuity  Managemerv 


Availability 

Management 


Service  Level 
Management 


STSM 


ITIL  —  CMMI  Correlation  Snapshot 


Service  Delivery 


RP 


SCO 
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DAR  RSKM 


The  Service  Desk 


Incident  Management 


Problem  Management 


Configuration 

Management 


STSU 


Control  processes  Planning  and 
M  implementing 

service  management 


Resolution  processes  Release  process 


new  or  changed 
services 


Service  delivery 
process 


Requirements  for  a 
management  systeri 


Relationship  processi 


ISO  20000  -  CMMI 
Correlation  Snapshot 


STSM 
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Business  Objectives 


Prioritizes  resources  and  projects  based  on  cost- 
benefit  considerations 


Manages  the  actual  development  process 

h 

Systems  must  be  designed  for  group-wide 
deployment 

Systems  and  functionality  are  reused  across 
products,  distribution  channels,  brands  and 
markets 


Systems  must  optimize  cross-organisational 
processes  and  make  it  possible  to  combine  parts  of 

i 

the  Group’s  products  into  new  products 


Business  Objectives 


By  reusing  system  elements,  even  across  different 
technical  platforms,  significant  efficiencies  are 
gained  in  the  development  of  systems 

Integrate  third  party  systems  into  the  whole 
system  complex,  regardless  of  the  technical 
deployment  platform 

Minimize  the  costs  associated  with  the  integration 
of  applications  and  tools  across  systems  and 
platforms 

Limited  but  adequate  set  of  market  leading 
technologies  are  used  as  standard  tools 


Business  Objectives 


SyDLC  and  TCO  must  be  implemented  in  such  a 
way  that  the  integrity  of  the  business  cannot  be 
compromised 

level  of  security  and  operations  must  be  high  and 
financially  sound 

Systems  and  platforms  must  have  a  high  quality 
level,  protecting  the  Group  against  errors,  down 
time,  security  breaches  and  data  loss 

Quality  level  must  correspond  with  risks, 
consequences  and  not  least  the  expectations  of  the 
customers 


Business  Objectives 

Systems  must  adhere  to  the  agreed  service  levels 
and  be  delivered  with  the  agreed  functionality 

Simple  and  accessible  user  interfaces,  adapted  to 
the  user’s  role  or  the  customer’s  needs 

Access  is  given  to  the  necessary  functionality  and 
information  from  the  underlying  business  system 
based  on  consolidated  data 

Systems  must  constantly  support  the  chosen  set  of 
distribution  channels  and  user  interfaces,  enabling 
the  Group  to  meet  the  customer  at  any  given  point 


Business  Objectives 


Knowledge  about  the  customers  and  their  behavior 
must  be  gathered  in  a  structured  way  on  each 
customer  interaction,  and  related  to  the  Group’s 
products 

Integrated  and  customer-facing  sales  and  advisory 
system  ensures  that  products  and  services  can  be 
developed  and  deployed  across  business  units, 
customer  segments  and  distribution  channels 

Reduce  the  Group’s  costs  by  optimizing  the  whole 
value  chain 


Costs  associated  with  the  rationalisation  of  processes  must 
be  minimal,  enabling  economically  feasible  automation  of 
even  small  business  processes 

Business  procedures  must  be  implemented  direct  as 
supported  processes,  guiding  employees  and  customers 
through  the  activities  with  as  little  prior  knowledge  as 
possible,  letting  them  concentrate  on  the  products  and 
actual  business. 

Enables  conversion  of  manual  activities  into  automatic 
sequences  without  changing  the  basic  design  of  the 
underlying  processes. 


Business  Objectives 


Business  Objectives 


Combine  activities  efficiently  and  flexibly  across 
distribution  channels,  partners,  brands  and  markets, 
wherever  this  is  desirable  from  a  business  point  of  view 

Systems  must  support  the  processes  which  gather, 
organize,  share  and  analyse  the  entire  knowledge  platform 
that  exists  about  customers,  products,  business  initiatives, 
organization,  employees,  etc 

Information  must  be  available  at  any  time  and  anywhere  to 
those  it  is  meant  for 

Group’s  management  processes  and  pricing,  they  must  be 
based  on  consolidated  and  sufficiently  current  data 


Business  Objectives 


Increase  flexibility  gradually  without  compromising  on 
efficiency  and  stability 

Diversity  is  handled  systematically  and  efficiently  by  using 
an  infrastructure,  which  efficiently  integrates  systems, 
processes  and  manual  activities  across  platforms  and 
technologies 

Infrastructure  is  provided  to  developers,  freeing  them  from 
having  to  programe  integration  and  flexibility  into  each 
system 

Use  of  market  leading  standards 

Design  of  system  elements  focusing  on  flexibility 


Business  Objectives 


System  elements  must  be  designed  to  scale  in  line  with 
business  growth  and  expansion 

System  elements  must  be  capable  of  handling  unexpected 
events 

Ensure  that  systems  can  continue  normal  operations  with 
the  least  impact  on  the  business 

Business  continuity  during  normal  operating  conditions  as 
well  as  in  disaster-like  situations 

Systems  design  must  if  possible  take  into  account  the 
changeability  of  externally  controlled  data  and  processes 

Readiness  for  change  by  implementing  changes  for  the 
entire  group 


Business  Objectives 


Resources  can  thus  be  reused  in  any  other  project  or  area 
in  the  Group  in  a  simple  and  efficient  way,  thereby 
ensuring  consolidation  of  both  data  and  functionality 

It  must  be  possible  to  combine  scattered  IT  resources  into 
complete  systems,  applications  and  actual  business 
processes 

Infrastructure  must  handle  the  coupling  dynamically  and 
parameterized 

Selection  of  coupling  method  must  not  be  based  on  a 
technology  choice  made  by  the  developers 


Business  Objectives 


Infrastructure  and  development  methods  must  as  a 
minimum  support  a  layering  of  systems  into  user 
interfaces,  business  logic  and  data 

Service  levels  must  if  possible  be  based  on  dynamic  and 
flexible  policies,  which  are  directly  definable  in  the 
operational  environment 

Infrastructure  must  efficiently  handle  error  detection  and 
quality  control  of  complete  system 

Infrastructure  must  efficiently  support  the  integration 

Architecture  is  an  essential  parameter  when  choosing  a 
third-party  system 
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Organizational 
Size  and  Functions 

Senior  manager 

15  ‘Division’  managers 

~80  Mid  Level  mangers 

~X00  project  /  program  /  acquisition  and 
line  mangers 

X,000  working  level  staff  relevant  to  the 
effort 

-800  Quality  related  personal 
-1000  ‘auditors’ 


Organizational  Background 
Size  and  Functions 


Acquisition 
Project  and  program  management 
In-house  full  development 
System  integration 
Service  units  (i.e.  IT,  Civil  Eng) 

HR 

Security  (Information  and  data) 
Facilities  and  infrastructure 
In-house  system  engineering 
Maintenance  and  support 
Web  centric  operational  architecture 


Organizational  Background 
ain  Related  Quality  Standards 

Internal  Quality  Standard 
EFQM 

CMMI  Suite  (SVC  /  ACQ  /  DEV) 

PMBOK  &  OPM3 
DoD  5000.01  &  5000.02 
ISO  14000 
OHAS  18000 
ITIL  V  3 
ISO  20000 
ISO  27001  &  27002 
ISO  9001 

Other  SEI  technologies  (RMM  /  P-CMM  /  TSP  /  PSP) 


IT  Quality  Management  Strategy 


L6  Strategic  Framework 
EFQM  /  Baldrige 


L5  Continuous  process 
improvements  (CMMI) 


L4  Organizational  (Cross  Units) 
QMS  Integration 


L3  Automation  (QMS  Application) 


L2  Processes  &  Implementation  of  best 
practices  &  standards  (ITIL) 


LI  Planning  &  Design  of  QMS  (based  on  ITIL  guidance 
and  IS09001:2000  preparation  and  certification) 


ent  System;  ITIL  =  Information  Technology  Infrastructure  Library;  ;  ISO  =  International  Standards  Organisation; 
rity  Model  Integration 


Sample  Improvement  Targets 

Service  reuse 

Improved  perception  and  response  time 
Interoperability 
Business  agility. 

Service  performance  and  its  impact  on  the 
organization  governance 


Cost  1  ^ 
Performance 
APT 


IPT  Structure 


Group  Level 


Overarching  IPT 
‘Division’ 


Contracting 
IPT 


Project  /  Task 
Management 
Environment 
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The  Approach  to  the  Solution 

Concept 

Best  practices  in  the  model  focus  on  activities  for 
providing  quality  services  to  the  customer  and  end 
users 


To  identify  improvement  targets  in  main  lifecycle 
areas  such  as  operations,  information,  governance, 
people  and  organizational  structure,  portfolios, 
project  execution,  and  finance 

Select  processes  that  are  critical  to  the  system 
success  such  as  stakeholder  management, 
technical  interfaces  and  integration 
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Requirements  OSSP  PIIDs  Affirmations  Total 

z 

4 

General 

Requirements 

0 

0  0  0 

!• 

^  Develop  Your  Quality 

Management  System  (QMS] 

0  0  0  0 

i 5 

4.1.1 

PsfaMsAjfour  o-rpapiraucps  & \$S 

0.00 

0.00 

0.00 

0.00 

s 

4.1.2 

£?peumepf  your  organisation's  QMS. 

0.00 

0.00 

0.00 

0.00 

W  7 

4.1.3 

dmpSemepf  your  organisation's  QMS. 

0.00 

0.00 

0.00 

0.00 

■■  B 

4.1.4 

A fatrtfatr?  your  organisation's  QMS. 

0.00 

0.00 

0.00 

0.00 

H - 

3  9 

h 

ill 

4.1.5 

Approve  your  organisation's  QMS. 

0.00 

0.00 

0.00 

0.00 

^  ^  Document  Your  Quality 
Management  System  (QMS] 

0  0  0  0 

4.2.1 

Manage  QOa&titAfaryapemept  System  Documents  0  0  0  0 

i  12 

4.2.1.1 

Develop  documents  for  your  organisation's  QMS. 

0.00 

0.00 

0.00 

0.00 

f  13 

4.2.1.2 

Make  sure  that  your  organisation's  QMS  documents  respect  and  reflect  what  you  do  and  how  you  do  it. 

0.00 

0.00 

0.00 

0.00 

fc 

4.2.2 

Prepare  Quality  A  •fapspemep,'  System  A  fartuaf 

WWW 

0  0  0 

0 

0.00 

L 

4.2.2.1 

Establish  a  quality  manual  for  your  organisation. 

0.00 

0.00 

0.00 

^  16 

4. 2. 2. 2 

Maintain  your  organisation's  quality  manual. 

0.00 

0.00 

0.00 

0.00 

bzi 

4.2.3 

PopfroJ  ORsaAO?  A  '/apapemepr  S^sfemDoc-umepfs 

WWW 

0  0  0 

0 

0.00 

u 

4.2.3.1 

Control  your  organisation's  QMS  documents. 

0.00 

0.00 

0.00 

i  i9 

4.2.3.2 

Control  documents  that  are  used  as  QMS  records. 

0.00 

0.00 

0.00 

0.00 

i 20 

4.2.4 

WWW 

EstabAsh  Quality  A'Japagemept  System  Records  0  0  0  0 

H  4  ►  H 


CM  Mis  SPs  Kobi  CM  Mis  GPs  ISO  9001  2008  ISO  9001  Sum  OHSAS  18001  2007  OHSAS  18001  2007  Sum  ISQ900Q-3  IS09000-3  Sum 
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□SSF  PIIDs 

Affirmation; 

Total 

3 

12 

12 

12 

12 

4 

4 

General  Requirements 

0 

0 

0 

0 

5 

4.1 

Develop  Vour  Quality  Management  System  (QMS) 

0 

0 

0 

0 

6 

4.2 

Document  Your  Quality  Management  System  (QMS) 

0 

0 

0 

0 

7 

S 

5 

Management  Requirements 

0 

0 

0 

0 

9 

5.1 

Show  Your  Commitment  to  Quality 

0 

0 

0 

0 

10 

5.2 

Focus  On  Your  Customers 

0 

0 

0 

0 

11 

5.3 

Support  Your  Quality  Policy 

0 

0 

0 

0 

12 

5.4 

Carry  Out  Your  QMS  Planning 

0 

0 

0 

0 

1 3 

5.5 

Allocate  QMS  Responsibility  and  Authority 

0 

0 

0 

0 

14 

5.6 

Perform  QMS  Management  Reviews 

0 

0 

0 

0 

15 

16 

6 

Resource  Requirements 

0 

0 

0 

0 

17 

6.1 

Provide  Required  QMS  Resources 

0 

0 

0 

0 

IS 

6.2 

Provide  Competent  QMS  Personnel 

0 

0 

0 

0 

13 

6.3 

Provide  Necessary  Infrastructure 

0 

0 

0 

0 

20 

6.4 

Provide  Suitable  Vork  Environment 

0 

0 

0 

0 

21  | 

22 

7 

Realization  Requirements 

0 

D 

D 

0 

23 

7.1 

Control  Product  Realisation  Planning 

0 

0 

0 

0 

24 

7.2 

Control  Customer-Related  Processes 

0 

0 

0 

0 

25 

7.3 

Control  Product  Design  and  Development 

0 

0 

0 

0 

26  |; 

7.4 

Control  Purchasing  and  Purchased  Products 

0 

0 

0 

0 

27  1 

7.5 

Control  Production  and  Service  Provision 

0 

0 

0 

0 

25 

7.6 

Control  Monitoring  and  Measuring  Equipment 

0 

0 

0 

0 

23 

30 

8 

Remedial  Requirements 
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Main  CMMI-SVC 


Strategic  Service  Management  (STSM) 

•  The  purpose  of  Strategic  Service  Management 
(STSM)  is  to  establish  and  maintain  standard 
services  in  concert  with  strategic  needs  and 
plans. 


Service  System  Development  (SSD) 

•  The  purpose  of  Service  System  Development 
(SSD)  is  to  analyze,  design,  develop,  integrate, 
verify,  and  validate  service  systems,  including 
service  system  components,  to  satisfy  existing 
or  anticipated  service  agreements 


Main  CMMI-SVC  PA 


Capacity  and  Availability  Management 
(CAM) 

•  The  purpose  of  Capacity  and  Availability 
Management  (CAM)  is  to  ensure  effective 
service  system  performance  and  ensure  that 
resources  are  provided  and  used  effectively  to 
support  service  requirements 


Main  CMMI-SVC  PA 


Decision  Analysis  and  Resolution  (DAR) 

•  The  purpose  of  Decision  Analysis  and 
Resolution  (DAR)  is  to  analyze  possible 
decisions  using  a  formal  evaluation  process  that 
evaluates  identified  alternatives  against 
established  criteria 
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Integrated  Project  Management  (IPM) 

•  The  purpose  of  Integrated  Project  Management 
(IPM)  is  to  establish  and  manage  the  project 
and  the  involvement  of  relevant  stakeholders 
according  to  an  integrated  and  defined  process 
that  is  tailored  from  the  organization’s  set  of 
standard  processes 


Main  CMMI-SVC  PA 


Incident  Resolution  and  Prevention  (IRP) 

•  The  purpose  of  Incident  Resolution  and 
Prevention  (IRP)  is  to  ensure  timely  and 
effective  resolution  of  service  incidents  and 
prevention  of  service  incidents  as  appropriate 
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Organizational  Process  Performance  (OPP) 

•  The  purpose  of  Organizational  Process 
Performance  (OPP)  is  to  establish  and  maintain 
a  quantitative  understanding  of  the  performance 
of  the  organization’s  set  of  standard  processes 
in  support  of  achieving  quality  and  process- 
performance  objectives,  and  to  provide  process- 
performance  data,  baselines,  and  models  to 
quantitatively  manage  the  organization’s 
projects 
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Service  Continuity  (SCON) 

•  The  purpose  of  Service  Continuity  (SCON)  is 
to  establish  and  maintain  plans  to  ensure 
continuity  of  services  during  and  following  any 
significant  disruption  of  normal  operations 

Service  Delivery  (SD) 

•  The  purpose  of  Service  Delivery  (SD)  is  to 
deliver  services  in  accordance  with  service 
agreements 


Second  Line  of  Use 
CMMI-SVC  PA 


Service  System  Transition  (SST) 

•  The  purpose  of  Service  System  Transition 
(SST)  is  to  deploy  new  or  significantly 
changed  service  system  components  while 
managing  their  effect  on  ongoing  service 
delivery 


Results  and  Benefits 


ethodology  conclusions. 


SME  related  conclusions  (preliminary). 


•  Small  units  have  better  acceptance  for  quality  activities 

•  Engineering  related  organizations  have  a  good  case  for  higher  performances 

•  Critical  dependency  must  be  mapped  in  the  OSSP. 

•  Benefits  for  organizations  operating  at  volatile  (uncertain)  segments. 


Model  formulation  is  complex,  but  evaluation  is  straightforward. 
Standard  tools,  and  the  mappings  make  it  easy  to  integrate  on  domains. 
Method  is  better  at  capturing  SMEs  insights  and  project  dynamics. 
Opportunity  to  further  include  additional  domains 
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Military  Combat  Services  Support 
Challenges  in  the  Battlefield  C4ISR  Systems 
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Extended  Systems 


Future  Plans 


Continue  refinement  of  model. 

•  Parameter  ranges. 

•  Roadmap  involving  multiple  lifecycles  and  external 
activities. 

•  Complete  Validation  and  Verification. 

Integrate  work  and  receive  feed  from  other 
initiatives. 

Continue  sharing  with  other  domains 


Discussion  Points 
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Cost  of  poor  planning 

Quantifying  the  operational  impact 

Effecting  and  effected  stakeholders 
mapping 

Quantifying  the  impact  of  support  planning 
on  the  development  teams 

Appling  this  model  on  other  domains 
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Introductions 


Instructor  Introduction 


Participant  Introductions 

(mechanics  depends  on  group  size  - 
individual  or  show  of  hands) 

•  name  (if  our  group  is  small  enough) 

•  company/position  -  or  type  of  company 
(government,  defense  industry, 
commercial  industry,  other) 

•  background  -  or  job  type  (manager, 
technical,  process  group,  other) 

•  software  architecture  background  / 
systems  architecture  background 
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Tutorial  Learning  Outcomes 


After  completing  this  half-day  tutorial,  attendees  should 

•  know  the  importance  of  architecture  to  the  achievement  of 
business,  product,  or  mission  goals 

•  know  that  quality  attributes  have  a  dominant  influence  on  a 
system’s  architecture 


•  be  familiar  with  essential  architecture-centric  engineering  activities 
and  some  example  methods 


•  know  how  to  specify  quality  attributes  meaningfully  through 
scenarios 


•  be  able  to  identify  where  architecture-centric  activities  and  work 
products  are  described  in  CMMI  VI. 3 

•  appreciate  how  to  interpret  the  new  architecture-centric  material  in 
CMMI  VI .3 


•  know  where  to  find  out  more  about  architecture-centric 
engineering  practices 
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What  Have  We  Learned  Over  the  Past  Year? 
Something  was  Missing. 


A  lot  of  people  overlooked  the  new  architecture  emphasis. 

•  Changes  in  the  model  that  explicitly  use  the  term 
“architecture”  are  in  the  informative  material  and  glossary. 

•  Those  who  understand  CMMI  may  overlook  the  new 
expectation  that  quality  attributes  are  an  equal  partner  to 
functionality. 

-Architecture  practices  thus  need  to  be  considered. 


Hence  we  decided  to  be  much  less  subtle  about 

architecture  in  the  title  and  we  have  allowed  more  time 
to  discuss  the  “whereabouts”  of  architecture  in  the  VI  .3 
model. 

Good  architecture  practices  are  essential  for  success 
with  CMMI  VI  .3! 
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Conventions  &  Caveats  for  the  Tutorial 


k 


The  coverage  of  architecture-centric  practices  in  CMMI  VI  .3  is  broad, 
focused  on  “products”  and  “solutions”  -  not  just  on  software. 

•  But  much  of  the  tutorial  material  came  from  SEI  assets  whose  focus  was 
software-intensive  systems.  Please  bear  this  in  mind.  We  believe  the 
principles  apply  beyond  simply  software. 

Our  focus  in  the  tutorial  will  be  on  CMMI  for  Development  because  that 
is  where  the  architecture-centric  practices  are  most  deeply  covered 
but  similar  changes  were  also  made  to  the  other  two  CMMI  models. 

CMMI  uses  the  term  “product”  to  refer  to  what  is  delivered  to  the 
customer  or  end-user.  In  this  tutorial,  we  will  often  use  the  term 
“system”  to  refer  to  the  product. 

This  tutorial  cannot  completely  convey  everything  you  might  like  to  learn 
about  architecture-centric  engineering. 

•  References  are  provided  at  the  end  for  you  to  learn  more. 
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Expected  Background  of  Participants 


Participants  must  have  an  understanding  of  the  basics  of  CMMI  models. 

•  This  tutorial  is  not  an  introduction  to  CMMI. 

•  It  is  not  a  substitute  for  VI  .3  Upgrade  Training. 

Familiarity  with  product,  system,  or  software  design  is  useful,  but  not 
required.  (Such  familiarity  will  be  of  benefit  to  you  in  the  hands-on 
exercises,  which  are  intended  to  give  you  a  grounding  in  some  key 
concepts.) 
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Presentation  Outline 


CMMI  VI. 3  -  Context  for  modern  engineering  practices  changes 

Introduction  to  Architecture 

Essential  Architecture  Practices 

Where  Are  the  Architecture-Centric  Practices  in  CMMI  VI  .3? 
Summary 

Questions  and  Answers 
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The  Problem:  CMMI  VI  .2  Did  Not  Adequately 
Cover  Modern  Engineering  Approaches 

Much  of  the  engineering  content  of  CMMI-DEV  VI  .2  was  ten  years  old. 

As  DEV  was  a  starting  point  for  the  other  two  constellations,  no  VI  .2 
model  adequately  addressed  modern  engineering  approaches  such 
as  architecture-centric  engineering. 

For  example,  both  RD  SG  3  and  RD  SP  3.2  emphasized  functionality 
and  not  non-functional  requirements. 

Also,  Engineering  and  other  PAs  rarely  mention  the  following  concepts: 

•  Quality  attributes 

•  Allocation  of  product  capabilities  to  release  increments 

•  Product  lines 

•  System  of  systems 

•  Technology  maturation  (and  obsolescence) 

•  Agile  methods 
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The  Solution:  Modernize  the  Engineering 

Content  in  CMMI  VI  .3 

The  slides  that  follow  portray  where  the  development  community  should 
be  today  relative  to  architecture-centric  practices  -  as  opposed  to 
how  they  were  portrayed  in  CMMI  VI  .2. 

Towards  the  end  of  today’s  half-day  tutorial,  we  will  revisit  how  CMMI 
Version  1 .3  addresses  these  and  other  modern  development 
approaches. 
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What  Is  an  Architecture? 


Informally,  an  architecture  is  the  blueprint  describing  the 
structure  of  a  system. 
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Architecture  is  Important 


The  quality  and  longevity  of  a  software-reliant  system  is  largely 
determined  by  its  architecture. 


In  recent  studies  by  OSD,  the  National  Research  Council,  NASA,  and 
the  NDIA,  architectural  issues  are  identified  as  a  systemic  cause  of 
software  problems  in  DoD  systems. 
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Why  Is  Architecture  Important? 


Represents  earliest 
design  decisions 


•  hardest  to  change 

•  most  critical  to  get  right 

•  communication  vehicle 
among  stakeholders 


First  design  artifact 
addressing 


performance 

modifiability 

reliability 

security 


The 

The 


Key  to  systematic  reuse 


Key  to  system  evolution 


•  transferable, 
reusable  abstraction 


•  manage  future  uncertainty 

•  assure  cost-effective  agility 


right  architecture  paves  the  way  for  system  success, 
wrong  architecture  usually  spells  some  form  of  disaster. 
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Architecture  is  About  Structure  and 


Structures  result  from  decisions 

•  Business  /  mission  goals  provide  a 
reasoned  basis  for  decisions. 

•  Each  decision  is  a  tradeoff  that 
enables  something  and  precludes 
other  things. 

•  Tradeoffs  are  driven  by  quality 
attribute  requirements. 

This  is  true  regardless  of  the  domain 
-  commercial  or  defense. 
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“Every  system  has  an  architecture... 


...encompassing  the  key  abstractions  and  mechanisms  that  define  that 
system's  structure  and  behavior...  In  every  case  -  from  idioms  to 
mechanisms  to  architectures  -  these  patterns  are  either 


intentional 

or 

accidental” 


-  Grady  Booch  in  the  Preface  to  Handbook  of  Software  Architecture 


F  Software  Engineering  Institute  CarnegieMellon 


Architecture  and  Strategy 


An  Intentional  Architecture  is  the 

embodiment  of  your  business  strategy 


•  Intentional  Architecture  links  technology 
decisions  to  business  goals 
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An  Accidental  Architecture 
limits  strategy  options 
•  Accidental  Architecture 
becomes  your  de  facto 
strategy 
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Class  Exercise  1 
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Presentation  Outline 


CMMI  VI  .3  -  Context  for  modern  engineering  practices  changes 
Introduction  to  Architecture 

Essential  Architecture  Practices 

Where  Are  the  Architecture-Centric  Practices  in  CMMI  VI  .3? 
Summary 

Questions  and  Answers 
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A  Warning  (PERMISSION  REQUESTED) 


“Architecture”  is  a  very  overloaded  word. 

•  All  the  good  words  are  taken. 

•  We  will  explain  some  common  uses  of  the  term  and  how  they  differ. 


CATBERT  EVIL  DIRECTOR 
OF  HUMAN  RESOURCES* 


IT 3  LIKE  TO  CHANGE 
MY  JOB  TITLE  TO 

something  WITH 

‘ARCHITECT*  IN  IT. 


©  Scott  Adams,  incJDi&L  by  UFS 


,  Inc. 


MY  DREAM  IS  TO 
DO  LESS  WORK  WHILE 
ALLEGEDLY  BEING 
MORE  VALUABLE. 
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Formal  Definition  of  Software  Architecture 


“  The  software  architecture  of  a  computing  system  is  the 
set  of  structures  needed  to  reason  about  the  system, 
which  comprise  software  components ,  relations  among 
them  and  properties  of  both.” 

Clements  et  al,  Documenting  Software  Architectures,  Second  Edition.  Addison-Wesley,  2011 
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Formal  Definition  of  System  Architecture 


A  system  architecture  describes  the  elements  and  interactions  of  a 
complete  system  including  its  hardware  elements  and  its  software 
elements. 

System  Architecture:  “The  fundamental  and  unifying  system  structure 
defined  in  terms  of  system  elements,  interfaces,  processes, 
constraints,  and  behaviors”'1 

Systems  Engineering  is  a  design  and  management  discipline  useful  in 
designing  and  building  large,  complex,  and  interdisciplinary  systems.2 


1  Rechtin,  E.  Systems  Architecting:  Creating  and  Building  Complex  Systems.  Englewood  Cliffs,  NJ  :  Prentice-Hall, 
1991. 

2  International  Council  On  Systems  Engineering  (INCOSE),  Systems  Architecture  Working  Group,  1996. 
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Implications 


Architecture  is  an  abstraction  of  a  system. 

Architecture  defines  the  properties  of  elements. 

Systems  can  and  do  have  many  structures. 

Every  software-intensive  system  has  an  architecture. 

Just  having  an  architecture  is  different  from  having  an  architecture  that 
is  known  to  everyone. 

If  you  don’t  develop  an  architecture,  you  will  get  one  anyway  - 

and  you  might  not  like  what  you  get! 
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Structures  and  Views  - 1 


One  house,  many  views 


Carpentry  view 
Plumbing  view 
Electrical  view 
Ductwork  view 


No  single  view  accurately  represents  the  house. 


No  single  view  can  be  used  to  build  the  house. 

Although  these  views  are  pictured  differently,  and  each  has 
different  properties,  all  are  related.  Together,  they  describe  the 
architecture  of  the  house. 
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Structures  and  Views  -  2 


A  human  body 
comprises  multiple 

structures. 


a  static  view  of 

one  human 
structure 


a  dynamic  view 

of  that  structure 


One  body  has  many  structures,  and  those  structures  have  many 
views.  So  it  is  with  software  and  systems. 
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Presentation  Outline 


CMMI  VI  .3  -  Context  for  modern  engineering  practices  changes 
Introduction  to  Architecture 

Essential  Architecture  Practices 

Where  Are  the  Architecture-Centric  Practices  in  CMMI  VI  .3? 
Summary 

Questions  and  Answers 
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What  is  Architecture-Centric  Engineering? 


Architecture-Centric  Engineering  (ACE)  is  the 

discipline  of  using  architecture  as  the  focal  point  for 
performing  ongoing  analyses  to  gain  increasing 
levels  of  confidence  that  systems  will  support  their 
missions. 

Architecture  is  of  enduring  importance  because  it  is 
the  right  abstraction  for  performing  ongoing  analyses 
throughout  a  system’s  lifetime. 


The  SEI  ACE  Initiative 

develops  principles,  methods, 
foundations,  techniques, 
tools,  and  materials  in 
support  of  creating,  fostering, 
and  stimulating  widespread 
transition  of  the  ACE 
discipline. 
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System  Development 


If  function  were  all  that 
mattered,  any  monolithic 
implementation  would  do, 

..but  other  things 
matter. . . 


The  important  quality  attributes  and  their  characterizations  are  key. 


Modifiability 

Interoperability 

Availability 

Security 

Predictability 

Portability 


analysis,  design,  development,  evolution 


Quality 
Attribute  Drivers 


Software  & 
System 
Architectures 


Software  & 
System 


) 


has  these  qualities 


The  Non-functional 
Requirements 
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Specifying  Quality  Attributes 


Quality  attributes  are  rarely  captured  effectively  in 
requirements  specifications;  they  are  often  vaguely 
understood  and  weakly  articulated. 

Just  citing  the  desired  qualities  is  not  enough;  it  is 
meaningless  to  say  that  the  system  shall  be  “modifiable” 
or  “interoperable”  or  “secure”  without  details  about  the 
context. 


,if** 


bcp033  04  fotosearch.com 


The  practice  of  specifying  quality  attribute  scenarios  can 
remove  this  imprecision  and  allows  desired  qualities  to 
be  evaluated  meaningfully. 

A  quality  attribute  scenario  is  a  short  description  of  an 
interaction  between  a  stakeholder  and  a  system  and  the 
response  from  the  system. 
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Parts  of  a  Quality  Attribute 


V  \ 


Vi  WA 

Artifact: 

) 

Stimulus 

L 

Process,  Storage, 

1 1 

JV  /w* 

Processor, 

W 

J 

Communication 

^  j 

Response 


SOURCE 


ENVIRONMENT 


RESPONSE 

MEASURE 
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Example  Quality  Attribute  Scenario 

A  “performance”  scenario:  A  remote  user  requests  a  data  base 
report  under  peak  load  and  receives  it  in  under  5  seconds. 


SOURCE 
Remote  user 


ENVIRONMENT 

Database  under 
peak  load 


RESPONSE 
MEASURE 
under  5 
seconds 
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Additional  Example  Scenarios 


Security:  “A  correctly  identified  individual  modifies  system  data  from  an 
external  site  incorrectly.  The  system  maintains  an  audit  trail  and  the 
correct  data  is  restored  within  one  day.” 


Modifiability:  “A  user  requests  a  change  to  the  user  interface;  The 
modification  is  made  by  a  developer  with  no  side  effects,  in  three 
hours.” 


Availability:  “An  unanticipated  external  message  is  received  by  a 
process  during  normal  operation.  The  process  informs  the  operator  of 
the  message’s  receipt,  and  the  system  continues  to  operate  with  no 
downtime.” 
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Class  Exercise  2 
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Architecture-Centric  Activities 


Architecture-centric  activities  include  the  following: 

•  creating  the  business  case  for  the  system 

•  understanding  the  requirements 

•  creating  and/or  selecting  the  architecture 

•  documenting  and  communicating  the  architecture 

•  analyzing  or  evaluating  the  architecture 

•  implementing  the  system  based  on  the  architecture 

•  ensuring  that  the  implementation  conforms  to  the  architecture 

•  evolving  the  architecture  so  that  it  continues  to  meet  business  and 
mission  goals 
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Some  SEI  Techniques,  Methods,  and  Tools 


creating  the  business  case  for  the  system 

understanding  the  requirements 

Quality  Attribute  Workshop  (QAW)  * 
Mission  Thread  Workshop  (MTW)  * 

creating  and/or  selecting  the  architecture 

Attribute-Driven  Design  (ADD) 
and  ArchE 

documenting  and 
communicating  the  architecture 

Views  and  Beyond  Approach;  AADL 

analyzing  or  evaluating  the  architecture 

Architecture  Tradeoff  Analysis  Method 
(ATAM)  *;  SoS  Arch  Eva!  *;  Cost  Benefit 
Analysis  Method  (CBAM);  AADL 

implementing  the  system  based  on  the 
architecture 

ensuring  that  the  implementation  conforms  to 
the  architecture 

ARMIN 

evolving  the  architecture  so  that  it  continues  to 
meet  business  and  mission  goals 

Architecture  Improvement  Workshop 
(AIW)*  and  ArchE 

ensuring  use  of  effective  architecture 
practices 

Architecture  Competence  Assessment 

*  — 


=  indicates  a  software  engineering  method  that  has  been  extended  to  systems  engineering 
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Building  the  Business  Case  for  the  System 


How  to  do  this  is  beyond  the  scope  of  this  tutorial. 

Some  common  business  /  mission  drivers  for  systems  include 

•  Reduce  total  cost  of  ownership 

•  Improve  capability/quality  of  system 

•  Improve  market  position 

•  Support  improved  business  processes 

•  Improve  confidence  in  and  perception  of  system 
Results  gleaned  from 

•  25  architecture  evaluations 

-  18  government  systems,  7  commercial  systems 

•  190  distinct  business  goals 


Kazman  &  Bass,  Categorizing  Business  Goals  for  Software  Architectures,  CMU/SEI-2005-TR-021 
http://www.sei.cmu.edu/reports/05tr021  .pdf 
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Understanding  the  Requirements  - 
The  SEI’s  Quality  Attribute  Workshop 

The  purpose  of  the  SEI  Quality  Attribute  Workshop  (QAW)  is  to  discover, 
early  in  the  life  cycle,  the  driving  quality  attribute  requirements  of  a 
software-intensive  system. 

QAW  Steps 

1.  QAW  Presentation  and  Introductions 

2.  Business/Programmatic  Presentation 

3.  Architectural  Plan  Presentation 

4.  Identification  of  Architectural  Drivers 

5.  Scenario  Brainstorming 

6.  Scenario  Consolidation 


Barbacci,  et  al.,  Quality  Attribute  Workshops  (3rd  Ed.),  CMU/SEI-2003-TR-016 
http://www.sei.cmu.edu/library/abstracts/reports/03tr016.cfm 


7.  Scenario  Prioritization 

8.  Scenario  Refinement 


An  Approach  to  Architecture  Creation 


The  Attribute-Driven  Design  (ADD)  method  is  an  approach  to  defining 
software  architecture  by  basing  the  design  process  on  the  quality 
attribute  requirements  of  the  system. 


Decompose 
according 
to  tactics. 


Architecture 
Design  Concept 


Architecture 

Describe 

Description 

decomposition. 
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Class  Exercise  3 
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Creating  the  Architecture 


How  to  do  this  is  beyond  the  scope  of  this  tutorial. 

Part  of  the  ADD  approach  is  to  pick  architectural  patterns  and  tactics 
that  address  particular  quality  attributes. 

Patterns  represent  a  packaging  of  a  number  of  design  decisions  we 
refer  to  as  tactics. 


Each  tactic  is  a  design  option  available  to  the  architect. 

A  pattern  typically  employs  several  different  tactics  to  promote  various 
quality  attributes. 

Example:  Tactics  to  influence  availability  (keep  faults  from  becoming 
errors)  include 

-  Fault  Detection 
Fault  Recovery 

-  Fault  Prevention 
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Summary  of  Availability  Tactics 


Fault 


Fault 

Detection 


•  Ping/Echo 

•  Heartbeat 

•  Exception 


Availability 


Fault 
Recovery 
Preparation 
and  Repair 


j 


Voting 

Active 

Redundancy 

Passive 

Redundancy 

Spare 


Fault  Recovery 
and 

Reintroduction 


j 


•  Shadow  Operation 

•  State 

Resynchronization 

•  Checkpoint/ 
Rollback 


Fault 

Prevention 


J 


Removal  from 

Service 

Transactions 

Process 

Monitor 


Fault 
masked 
or  repair 
made 
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Other  Tactics 


There  are  tactics  for 

•  modifiability 

•  performance 

•  security 

•  testability 

•  usability 


See  Software  Architecture  in  Practice  for  a  more  complete  treatment  of 
the  subject. 
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Analyzing  the  Architecture  -  SEI’s  Architecture 
Tradeoff  Analysis  Method®  (ATAM®) 

The  ATAM  is  an  architecture  evaluation  method  that  focuses  on  multiple 
quality  attributes. 


Business 

Drivers 

Quality 

Attributes 

Scenarios 

Software 
|  Architecture 

Architectural 

Approaches 

Architectural 
Decisions  | 

h 


Y 

\ 

A 

analysis 

; 

K 

/ 

impacts 


distilled 

into 


Risk  Themes 


Tradeoffs 


Sensitivity  Points 


Non-Risks 


Risks 
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ATAM  Phases 


ATAM  evaluations  are  conducted  in  four  phases. 


Duration:  varies 
Meeting:  primarily 
phone,  email 


Duration:  1.5-2  days  each  for  Duration:  varies 
Phase  1  and  Phase  2  Meeting:  primarily 

Meeting:  typically  conducted  phone,  email 
at  customer  site 
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ATAM  Evaluative  Phases  (1  &  2) 


1.  Present  the  ATAM 

2.  Present  business  drivers 

3.  Present  architecture 

4.  Identify  architectural  approaches 

5.  Generate  quality  attribute  utility  tree 

6.  Analyze  architectural  approaches 

7.  Brainstorm  and  prioritize  scenarios 

8.  Analyze  architectural  approaches 

9.  Present  results 


Presentation 


Investigation 
and  Analysis 

Testing 

Reporting 


Phase  2  =  Recap  of  Phase  1  plus 
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Phase  1 


Documenting  the  Architecture 


Architecture  documentation  establishes  the  set  of  design  decisions  that 
must  be  made  along  the  way  to  establishing  and  maintaining  the 
architecture. 

An  architecture  is  a  multidimensional  construct,  too  involved  to  be  seen 
all  at  once. 

Recall:  systems  are  composed  of  many  structures. 

A  view  is  a  representation  of  a  structure. 

We  use  views  to  manage  complexity  by  separating  concerns. 


Three  Types  of  Views 


Different  types  of  views  show  different  types  of  information: 

1.  Module  views  show  how  the  system  is  structured  as  a  set  of  code  units. 

2.  Component-and-connector  views  show  how  the  system  is  structured  as  a 
set  of  elements  with  runtime  behaviors  and  interactions. 

3.  Allocation  views  show  how  the  system  relates  to  non-software  structures  in 
its  environment. 

Every  view  contains  information  from  at  least  one  of  these  categories. 

Some  views  contain  information  from  more  than  one  category,  but  these 
are  often  difficult  to  understand. 
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View-Based  Documentation 


Views  give  us  our  basic  principle  of  architecture  documentation 


Architecture 
for  System 
XYZ 


Documentation 
beyond  views 


Documenting  an  architecture  is  a  matter  of  documenting  the  relevant  views, 
and  then  adding  documentation  that  applies  to  more  than  one  view. 


The  choice  of  views  used  depends  on  the  nature  of  the  system 
and  the  stakeholder  needs. 
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Software  Architecture  Documentation  Needs 


Runtime  views  to  show  how  software  will  handle: 

•  hazards,  faults,  and  errors 

•  fault  tolerance/reconfigurations 

•  performance 

•  data  (e.g.,  quality,  timeliness,  ownership,  access  privileges) 

•  interface  boundaries 

Non-runtime  views  of  software  (vital  to  project  planning,  allocating  work 
assignments,  designing  for  modifiability,  reusability,  portability, 
extensibility,  etc.,  facilitating  incremental  development,  and  a  host  of 
other  critical  purposes) 

Architectural  decisions  and  the  rationale/implications/impact  of  those 
decisions  on  key  system  qualities 
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Implementing  and  checking  conformance 


Press  on  to  implementing  the  system  in  accordance  with  the 
architecture. 

Have  processes  and  supporting  tools  to  check  for  conformance  with  the 
architecture. 

Unfortunately,  a  lot  of  this  work  today  is  often  not  automated. 
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So  How  Well  Does  This  Work? 

Study:  Impact  of  Army  Architecture  Evaluations 

Twelve  Army  programs  that  had  conducted  ATAM  or  QAW  exercises  in 
a  study  to  elicit  the  perceived  impact  the  ATAM  evaluations  and  QAWs 
had  on  system  quality  and  the  practices  of  the  acquisition  organization. 

Results  showed 

•  6/12:  cost  less  than  or  equal  to  traditional  techniques 

•  1 0/1 2:  quality  of  results  greater  than  or  equal  to  traditional 
techniques 

•  10/12:  helped  understand  and  control  cost  and  schedule 

•  1 2/1 2:  increased  understanding  of  system’s  quality  attribute 
requirements,  design  decisions,  and  risks 

•  1 2/1 2:  good  mechanism  for  communication  among  stakeholders 

•  8/12:  improved  the  architecture 


The  context  of  use  had  a  significant  impact  on  the  results  enjoyed. 
Architecture-centric  acquisition  is  key  to  reaping  maximal  benefit. 
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Architecture  Practices  are  Having  an  Impact  iof2 

Results  of  2008  survey  of  12  Army  projects  that  employed  ATAM/QAW2 


Artifact  Improvement 


Minimal  Moderate  Significant  Very  Substantial 

■  Quality  Attributes  ■  Architecture  a  Risks 


•  Most  reported  significant 
improvement  in  their 
architecturally-significant 
artifacts 

•  Architecture  teams  were 
able  to  achieve 
understanding  of 
stakeholder  expectations 
and  the  implications  of 
architectural  decisions  on 
user  needs 


2  Source:  Impact  of  Army  Architecture  Evaluations,  CMU/SEI-2009-SR-007 
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Architecture  Practices  are  Having  an  Impact  2  of 2 


Results  of  2008  survey  of  12  Army  projects  that  employed  ATAM/QAW 

•  Majority  reported  very 
substantial  or  significant 
improvement  in  stakeholder 
communication 


•  Stakeholders,  collectively, 
are  able  to  achieve  a 
common  understanding  of 
the  system  under 
development 

-  Increases  likelihood  that 
product  will  address 
expectations/user  needs 

-  Improves  chances  for 
program  success 


Communication  Improvement 


Number  of  Programs 
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What  About  System  of  Systems  (SoS)?  - 1 


The  software-intensive  systems  in  an  SoS  are  likely  to  have  been 
developed  independently  of  each  other. 


Severe  integration  and  runtime  problems  thus  arise  due  to 
inconsistencies  in  how  quality  attributes  are  addressed. 

Thus,  architecture  is  even  more  important  in  an  SoS  context,  not  less. 
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What  About  System  of  Systems  (SoS)?  -  2 


Therefore,  in  an  SoS  context: 


•  Each  software-intensive  system 
in  an  SoS  has  its  own 
architecture  that  addresses  both 
system  and  software  aspects. 

•  The  SoS  itself  has  an 
architecture  whose  elements 
are  the  architectures  of  the 
individual  software-intensive 
systems. 

A  uniform  approach  for  specifying 
quality  attribute  requirements 
and  evaluating  SoS  and 
software-intensive  system 
architectures  against  such 
requirements  is  thus  needed. 
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The  Need  for  Augmented  Mission  Threads  in 
DoD  SoS  Architecture  Definition 

DoDAF  is  the  SoS  architecture  framework  for  the  DoD. 

•  It  provides  a  good  set  of  architectural  views  for  an  SoS 
architecture. 

•  It  inadequately  addresses  cross-cutting  quality  attribute 
considerations. 

System  use  cases  focus  on  a  functional  slice  of  the  system. 

More  than  DoDAF  and  system  use  cases  are  needed  to  ensure  that  the 
SoS  architecture  satisfies  its  end-to-end  functional  requirements  and 
quality  attribute  needs. 

SoS  end-to-end  mission  (operational  or  user)  threads  augmented  with 
quality  attribute  considerations  are  needed  to  help  develop,  and  later 
evaluate,  the  SoS  architecture. 
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One  Approach 


SEI  developed  and  applied  a  two-pronged  approach  to  address  the  early 
identification  of  quality  attribute  inconsistencies,  ambiguities,  and 
omissions  within  system  and  SoS  architectures  (in  Directed  and 
Acknowledged  SoS  contexts). 

1.  Perform  a  "first  pass"  identification  of  inconsistencies,  ambiguities,  and 
omissions  across  the  constituent  systems,  at  the  SoS  level,  using  end- 
to-end  mission  threads  that  are  augmented  with  quality  attribute 
concerns  from  SoS  stakeholders. 

The  approach  involves  a  series  of  workshop  and  evaluations. 

Mission  Thread  Workshop 
Architecture  Challenge  Workshop 
SoS  Architecture  Evaluation 

2.  Constituent  systems  that  are  “problematic”  are  further  evaluated  using 
the  system  and  software  architecture  evaluation  method  (based  on  the 
ATAM),  using  the  augmented  mission  threads  from  the  Mission  Thread 
Workshops. 

System  and  Software  ATAM 
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SoS  and  Quality  Attribute  Elicitation, 
Specification,  and  Analysis 


SoS 
Architecture 
/aluations 


Systems 

ATAMs 


System  and  Software 
Architectures  Risks 


Vignettes 
Mission  Threads? 
Sos  Architecture 
Plans 


SoS  Mission/ 
Business  Drivers 
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Architectural  Reuse 


An  architecture  represents  a  significant  investment. 

Why  use  it  for  only  one  system? 

Most  organizations  produce  families  of  similar  systems,  differentiated  by 
features. 


Software  Engineering  Institute  Carnegie  Mellon 


Architecture  and  CMMI  VI  .3 

©2011  Carnegie  Mellon  University 


57 


Software  Product  Lines 


A  software  product  line  is  a  set  of  software-intensive  systems  sharing  a 
common,  managed  set  of  features  that  satisfy  the  specific  needs  of  a 
particular  market  segment  or  mission  and  that  are  developed  from 
common  set  of  core  assets  in  a  prescribed  way. 
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Successful  Software  Product  Lines 


Improvements  in  cost,  time  to  market,  and  productivity  that  come  with 
successful  product  lines  abound. 

•  Cummins  reduced  the  time  it  takes  to  produce  software  for  a  diesel  engine 
from  one  year  to  one  week. 

•  Motorola  realized  a  400%  productivity  improvement  in  a  family  of  one-way 
pagers. 

•  Hewlett-Packard  reduced  time  to  market  by  a  factor  of  seven  and  increased 
productivity  by  a  factor  of  four  in  a  family  of  printers. 

•  The  NRO  built  a  ground  control  system  with  10%  of  the  expected  number  of 
developers  and  reduced  defects  by  90%. 

•  Nokia  reports  producing  25  to  30  different  phone  models  per  year  by  using  a 
product  line  approach. 
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Core  asset  base 


Building  the  Core  Asset  Base 


Core  assets  include: 


Requirements  Domain 

and  model 

requirements 
analysis 


Software 

architecture 


Test  plans, 
test  cases, 
and  data 


People 
knowledge 
and  skills 


Processes, 
methods,  and 
tools 


Performance  Documentation 
engineering 


...with  attached  processes 


Budgets, 
schedules, 
work  plans 
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Core  asset  base 


Building  a  Product 


d 


I 


Requirements 

and 

requirements 

analysis 


Test  plans, 
test  cases, 
and  data 


Domain 

model 


Software  Performance  Documentation 

architecture  engineering 


People  Processes,  Budgets,  ...and 

knowledge  methods,  schedules,  software 

and  skills  and  tools  work  plans 


=  product-specific 
element 

=  product-specific  non¬ 
software  assets 
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Core  asset  base 


Building  Subsequent  Products 


I 


Requirements 

and 

requirements 

analysis 


Test  plans, 
test  cases, 
and  data 


Domain 

model 


Software  Performance  Documentation 

architecture  engineering 


People  Processes,  Budgets,  ...and 

knowledge  methods,  schedules,  software 

and  skills  and  tools  work  plans 


=  product-specific 
element 

=  product-specific  non¬ 
software  assets 
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Feedback 


Product  #n 


J 


Requirements 

and 

requirements 

analysis 


Test  plans, 
test  cases, 
and  data 


Domain 

model 


People 
knowledge 
and  skills 


Software 

architecture 


Processes, 
methods, 
and  tools 


Performance 

engineering 


Budgets, 
schedules, 
work  plans 


Documentation 


...and 

software 


w 


~  Software  Engineering  Institute  Carnegie  Mellon 


Architecture  and  CMMI  VI  .3 

©2011  Carnegie  Mellon  University 


63 


Widespread  Application  - 1 


akvasmart}© 


Asea  Brown  Boveri 


jIXISA 

COMMUNICATIONS 


Feed  control  and  farm 
management  software 


Bold  Stroke  Avionics 


E-COM  Technology  Ltd. 


Gas  turbines,  train  control, 
semantic  graphics  framework 


Internet  payment  gateway 
infrastructure  products 


ERICSSON  ^ 


Computer  printer  servers, 
storage  servers,  network  camera 
and  scanner  servers 


Customized  solutions  for 
transportation  industries 


Medical  imaging  workstations 


li  n  v  *  if  t 


Firmware  for  computer 
peripherals 


O 

Lucent  Technpfagres 

EH  LdtA  liNVJd'UTi 


5ESS  telecommunications 
switch 


AXE  family  of 

telecommunications  switches 


Software  for  engines, 
transmissions  and 
controllers 


©LG 

Elevator  control  systems 


LSI 


IjOUC 


RAID  controller  firmware 
for  disk  storage  units 


NOKIA 

Mobile  phones,  mobile  browsers,  telecom 
products  for  public,  private  and  cellular 
networks 


Interferometer  product  line 
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Widespread  Application  -  2 


PHILIPS 


ROM  B0SCH  ® 


High-end  televisions, 

PKI  telecommunications  switching  system, 
diagnostic  imaging  equipment 

Rockwell 

Collins 

Commercial  flight  control  system  avionics, 
Common  Army  Avionics  System  (CAAS), 
U.S.  Army  helicopters 


Office  appliances 


s  A  L  i  O  N' 

TM6«T  WPM.  PfllVER 

Revenue  acquisition 
management  systems 


TGLVGNT 


Automotive  gasoline  systems 

SIEMENS 

Software  for  viewing  and  quantifying 
radiological  images 


Climate  and  flue  gas 
measurement  devices 


Symbian 

EPOC  operating  system 


Test  range  facilities 


Industrial  supervisory  control 
and  business  process 
management  systems 


mme 


Command  and  control 
simulator  for  Army  fire 
support 


CUtel  JS 


FIDELITY 


■ATIQ4J  «.i  riHANClik' 


Support  software 


MOTOROLA 


Pagers  product  line 
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Presentation  Outline 


CMMI  VI  .3  -  Context  for  modern  engineering  practices  changes 
Introduction  to  Architecture 

Essential  Architecture  Practices 

Where  Are  the  Architecture-Centric  Practices  in  CMMI  VI  .3? 

Summary 

Questions  and  Answers 
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Modern  Engineering  Approaches  in  CMMI  - 1 


For  Version  1.3,  CMMI  provides  better  coverage  of  architecture¬ 
centric  practices  (mostly  by  changes  to  informative  material): 

•  creating  the  business  case  for  the  system  (partially  in  RD  and  TS) 

•  understanding  the  requirements  (RD) 

•  creating  and/or  selecting  the  architecture  (TS) 

•  documenting  and  communicating  the  architecture  (RD,  TS) 

•  analyzing  or  evaluating  the  architecture  (RD,  TS,  VAL,  VER) 

•  implementing  the  system  based  on  the  architecture  (TS;  A/PL  notes) 

•  ensuring  that  the  implementation  conforms  to  the  architecture  (VER) 

•  evolving  the  architecture  so  that  it  continues  to  meet  business  and 
mission  goals  (implicit  in  the  phrase  “establish  and  maintain”) 


RD  =  Requirements  Development  PA  TS  =  Technical  Solution  PA 

VER  =  Verification  PA  VAL  =  Validation  Pa 

A/PL  =  Agile  and  Product  Lines  notes  (primarily  in  the  Introductory  Notes  of  a  PA) 
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Modern  Engineering  Approaches  in  CMMI  -  2 


CMMI  VI  .3  also  provides  an  improved  terminology  to  support 
understanding  and  use  of  architecture-centric  practices 

•  Updated  the  glossary  to  include  new  terms  (and  modified  some  old  terms) 

•  Updated  the  informative  material  (especially  ARD  and  ATM  in  ACQ;  RD,  TS, 
and  VER  in  DEV;  and  SSD  in  SVC)  to: 

-  make  use  of  the  new  terms 

-  bring  more  emphasis  to  quality  attributes  and  thus  strike  a  better  balance 
between  functional  and  non-functional  requirements 

•  Replaced  selected  uses  of  overloaded  terms  such  as  “performance”  with  an 
appropriate  qualifying  phrase. 
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Modern  Engineering  Approaches  in  CMMI  -  3 


Added  and  revised  the  informative  material  throughout  the  Engineering 
PAs  (in  particular)  to  appropriately  mention  the  following  engineering 
concepts: 

•  quality  attributes  (i.e.,  non-functional  requirements  or  “ilities”) 

•  architecture-centric  practices 

•  product  lines,  system  of  systems 

•  allocation  of  product  capabilities  to  release  increments 

•  technology  maturation  (and  obsolescence) 

These  concepts  are  mentioned  in  example  boxes,  in  examples  provided 
in  the  notes,  and  in  discussions  that  mention  various  approaches  that 
can  be  used. 

When  functional  requirements  are  discussed,  mention  of  quality 
attributes  is  added  to  balance  the  view  of  requirements. 
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Requirements  Development 


SG  1 :  Develop  Customer  Requirements 
SP1.1  Elicit  Needs 

SP  1 .2  Transform  Stakeholder  Needs  into 

[Prioritized]  Customer  Requirements 
SG  2:  Develop  Product  Requirements 


In  SP1 .2,  added  that  customer 
requirements  should  be  prioritized 
based  on  their  criticality  to  the 
customer  and  other  stakeholders 
“representing  all  phases  of  the 
product's  lifecycle  ...  including 
business  as  well  as  technical 
functions.” 


SP  2.1 

SP  2.2 
SP  2.3 


Establish  Product  and  Product  Component 
Requirements 

Allocate  Product  Component  Requirements 
Identify  Interface  Requirements 


In  SP  2.1,  added  a  focus  on 

architectural  requirements  and  quality 
attribute  measures. 

In  SP  2.2,  added  a  subpractice 
allocating  requirements  to  delivery 
increments. 


SG  3:  Analyze  and  Validate  Requirements 

SP  3.1  Establish  Operational  Concepts  and 
Scenarios 


Addressed  “Quality  attributes”  (QAs)  as 
well  as  functionality  in  SG3  and  SP  3.2 
statements. 


SP  3.2 

SP  3.3 
SP  3.4 
SP  3.5 


Establish  a  Definition  of  Required  In  SP  3.1,  broadened  emphasis  to 

Functionality  and  Quality  Attributes  “operational,  sustainment,  and 

development  scenarios. 

Analyze  Requirements  In  SP  3.2,  determined  architecturally- 

Analyze  Requirements  to  Achieve  Balance  significant  qas  from  mission  and 

business  drivers. 

Validate  Requirements 
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Technical  Solution 


SG  1 :  Select  Product  Component  Solutions 

SP  1 .1  Develop  Alternative  Solutions  and 
Selection  Criteria 

SP  1 .2  Select  Product  Component  Solutions 
SG  2:  Develop  the  Design 

SP  2.1  Design  the  Product  or  Product 
Component 

SP  2.2  Establish  a  Technical  Data  Package 
SP  2.3  Design  Interfaces  Using  Criteria 
SP  2.4  Perform  Make,  Buy,  or  Reuse  Analyses 
SG  3:  Implement  the  Product  Design 

SP3.1  Implement  the  Design  s 

SP  3.2  Develop  Product  Support  Documentation 


Intro  Notes:  “QA  models, 
simulations,  prototypes  or  pilots 

can  be  used  to  provide  additional 
information  about  the  properties  of  the 
potential  design  solutions  to  aid  in  the 
selection  of  solutions.  Simulations  can 
be  particularly  useful  for  projects 
developing  systems-of-systems.” 

In  SP  1.1,  Added  an  example 
selection  criterion,  “Achievement  of  key 
quality  attribute  requirements”  and  a 
new  subpractice:  “Identify  re-usable 
solution  components  or  applicable 
architecture  patterns.”. 

In  SP  2.1,  described  architecture 
definition  tasks  such  as  selecting 
architectural  patterns  and  formally 
defining  component  behavior  and 
interactions  using  an  architecture 
description  language. 

In  SP  2.2,  added  subpractice  to 
determine  views  to  document 
structures  and  address  stakeholder 
concerns. 

In  SP  2.3,  mentioned  exception  and 
error  handling, 
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Product  Integration 


SG  1 :  Prepare  for  Product  Integration 

SP  1.1  Establish  an  Integration  Strategy 
SP  1.2  Establish  the  Product  Integration  Environment 

SP  1.3  Establish  Product  Integration  Procedures  and 
Criteria 

SG  2:  Ensure  Interface  Compatibility 


Revised  the  purpose  to  ensure 
proper  behavior  instead  of  proper 
function,  thereby  more  implicitly 
including  quality  attributes  as  well  as 
functionality. 


Changed  emphasis  from 
integration  sequence  to  an  emphasis 
on  integration  strategy,  i.e.,  the 
approach  to  receiving,  assembling, 
and  evaluating  product  components. 
The  architecture  will  significantly 
influence  the  selection  of  a  product 
integration  strategy. 


SP  2.1  Review  Interface  Descriptions  for 
Completeness 

SP  2.2  Manage  Interfaces 
SG  3:  Assemble  Product  Components  and  Deliver  the 


In  the  PA  notes,  addressed: 
interfaces  to  data  sources  and 
middleware;  APIs,  automated  builds, 
continuous  integration 


Product 


SP  3.1  Confirm  Readiness  of  Product  Components 
for  Integration 

SP  3.2  Assemble  Product  Components 

SP  3.3  Evaluate  Assembled  Product  Components 

SP  3.4  Package  and  Deliver  the  Product  or  Product 
Component 
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Validation 


SG  1 :  Prepare  for  Validation 

SP  1 .1  Select  Products  for  Validation 

SP  1 .2  Establish  the  Validation  Environment 

SP  1.3  Establish  Validation  Procedures  and 

Criteria 

SG  2:  Validate  Product  or  Product  Components 
SP2.1  Perform  Validation 

SP  2.2  Analyze  Validation  Results 


Reinforced  when  validation  occurs  in 
the  product  lifecycle:  “validation  is 
performed  early  (concept/exploration 
phases)  and  incrementally  throughout 
the  product  lifecycle  (including 
transition  to  operations  and 
sustainment).” 

In  VAL  SP  1.1,  included  access 
protocols  and  data  interchange 
reporting  formats  as  examples  of  what 
to  validate. 

Also,  included  incremental  delivery 
of  working  and  potentially 
acceptable  product  as  an  example 
validation  method. 
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Verification 


SG  1 :  Prepare  for  Verification 

SP  1 .1  Select  Work  Products  for  Verification 
SP  1 .2  Establish  the  Verification  Environment 

SP  1 .3  Establish  Verification  Procedures  and 
Criteria 

SG  2:  Perform  Peer  Reviews 
SP  2.1  Prepare  for  Peer  Reviews 
SP  2.2  Conduct  Peer  Reviews 

SP  2.3  Analyze  Peer  Review  Data 

SG  3:  Verify  Selected  Work  Products 
SP3.1  Perform  Verification 

SP  3.2  Analyze  Verification  Results 


In  SP  1.1,  added  example  verification 
methods:  software  architecture 
evaluation  and  implementation 
conformance  evaluation  and 

continuous  integration. 

In  SP  1.3,  added  example  sources 
of  verification  criteria: 
customers  reviewing  work  products 
collaboratively  with  developers. 


In  SP  2.1 ,  added  example  type  of  peer 
review:  architecture  implementation 
conformance  evaluation 


In  SP  2.3,  added  examples  of  peer 
review  data  that  can  be  analyzed: 

user  stories  or  case  studies 
associated  with  a  defect  and  the 
end-users  and  customers  who  are 
associated  with  defects 
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Addressing  Agile  - 1 


Changes  Supporting  Use  of  Agile  Methods 

Because  CMMI  practices  are  written  for  use  in  a  broad  variety  of 
contexts,  business  situations,  and  application  domains,  it  is  not 
possible  (even  if  it  were  appropriate)  to  advocate  any  specific 
implementation  approach. 

However,  Agile  methods  and  approaches  are  now  in  wider  use,  and  so 
for  VI  .3,  it  seemed  appropriate  to  acknowledge  this,  identify  how 
Agile  approaches  can  address  CMMI  practices  and  conversely, 
identify  the  value  that  CMMI  can  bring  to  Agile  implementations. 

The  next  set  of  slides  describe  how  CMMI  VI  .3  addresses  Agile 
methods. 
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Addressing  Agile  -  2 


The  Problem 

Developers  that  use  Agile  methods  sometimes  resist  using  CMMI 
because  they  can’t  see  how  CMMI  practices  can  complement  or 
improve  the  effectiveness  of  Agile  methods. 

Overview  of  Solution 

Added  guidance  to  the  appropriate  PAs  to  do  the  following: 

•  Help  users  interpret  the  practices  in  a  context  where  Agile  methods 
are  used 

•  Reinforce  the  applicability  of  the  practices  in  an  Agile  environment 

•  Send  the  message  that  CMMI  is  a  robust  best  practice  framework 
meant  to  be  used  in  Agile  environments  as  well  as  other  development 
environments 
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Addressing  Agile  -  3 


Solution 

Added  a  new  section  to  DEV  Chapter  5  entitled  “Interpreting  CMMI 
When  Using  Agile  Approaches” 

•  This  section  describes  how  CMMI  practices  can  apply  in  a  variety  of 
development  environments.  It  also  describes  the  interpretive 
guidance  that  has  been  added  to  selected  PAs  for  use  in  Agile 
environments. 

Added  interpretive  guidance  to  the  following  PAs: 

•  In  DEV:  CM,  REQM,  PP,  RD,  TS,  PI,  VER,  PPQA,  and  RSKM 

•  In  ACQ:  AM,  ATM,  PMC,  and  PP 

•  In  SVC:  SSD 


Added  in  DEV  and  SVC  (SSD  only)  Agile-related  examples  as  bullets  in 
example  boxes  (informative  material). 
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Addressing  Agile  -  4 


A  note  added  in  the  RD  Intro  Notes: 

In  Agile  environments,  requirements  are  communicated  and  tracked  through 
mechanisms  such  as  product  backlogs,  story  cards,  and  screen  mock-ups. 
[snip]  Traceability  and  consistency  across  requirements  and  work  products  is 
addressed  through  the  mechanisms  already  mentioned  as  well  as  during 
start-of-iteration  or  end-of-iteration  activities  such  as  “retrospectives”  and 
“demo  days.”  [Emphasis  added] 

A  note  added  in  the  TS  Intro  Notes: 

In  Agile  environments,  the  focus  is  on  early  solution  exploration.  By  making 
the  selection  and  tradeoff  decisions  more  explicit,  the  Technical  Solution 
process  area  helps  improve  the  quality  of  those  decisions,  both  individually 
and  over  time,  [snip]  When  someone  other  than  the  team  will  be  working  on 
the  product  in  the  future,  release  information,  maintenance  logs,  and  other 
data  are  typically  included  with  the  installed  product.  To  support  future 
product  updates,  rationale  (for  trade-offs,  interfaces,  and  purchased  parts)  is 
captured  so  that  why  the  product  exists  can  be  better  understood,  [snip] 
'Emphasis  added] _ 
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Addressing  Product  Lines  - 1 


Likewise,  notes  have  been  added  to  the  Intro  Notes  of  selected  PAs 
to  explain  how  the  PA  can  be  effectively  applied  in  a  product  line 
environment. 
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Addressing  Product  Lines  -  2 


An  example  of  a  note  added  in  the  RD  Intro  Notes: 

For  product  lines,  engineering  processes  (including  requirements 
development)  may  be  applied  to  at  least  two  levels  in  the  organization.  At  an 
organizational  or  product  line  level,  a  “commonality  and  variation  analysis”  is 
performed  to  help  elicit,  analyze,  and  establish  core  assets  for  use  by  projects 
within  the  product  line.  At  the  project  level,  these  core  assets  are  then  used 
as  per  the  product  line  production  plan  as  part  of  the  project’s  engineering 
activities.  [Emphasis  added] 

An  example  of  a  note  added  in  the  TS  Intro  Notes: 

For  product  lines,  these  practices  apply  to  both  core  asset  development  (i.e., 
building  for  reuse)  and  product  development  (i.e.,  building  with  reuse).  Core 
asset  development  additionally  requires  product  line  variation  management 
(the  selection  and  implementation  of  product  line  variation  mechanisms)  and 
product  line  production  planning  (the  development  of  processes  and  other 
work  products  that  define  how  products  will  be  built  to  make  best  use  of  these 
core  assets).  [Emphasis  added] 
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Changes  in  CMMI  Terminology  - 1 


Allocated  requirement 

DEFINITION 

Requirement  that  Jeviesresults  from  levying  all  or  part  of  the  performance  and 
functionality  of  a  higher  level  requirement  on  a  lower  level  architectural 
element  or  design  component. 

More  generally,  requirements  can  be  allocated  to  other  logical  or  physical 
components  including  people,  consumables,  delivery  increments,  or  the 
architecture  as  a  whole,  depending  on  what  best  enables  the  product  or 
service  to  achieve  the  requirements. 


The  improvements  to  the  above  definition  make  the  substance  of  the  solution 
space  and  the  allocation  of  requirements  to  it  more  explicit,  enabling  insightful 
analyses  (including  verification)  of  requirements,  architectures,  and 
implementations. 
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Changes  in  CMMI  Terminology  -  2 


Architecture 

DEFINITION 

The  set  of  structures  needed  to  reason  about  a  product.  These  structures  are 
comprised  of  elements,  relations  among  them,  and  properties  of  both. 

In  a  service  context,  the  architecture  is  often  applied  to  the  service  system. 

Note  that  functionality  is  only  one  aspect  of  the  product.  Quality  attributes, 
such  as  responsiveness,  reliability,  and  security,  are  also  important  to  reason 
about.  Structures  provide  the  means  for  highlighting  different  portions  of  the 
architecture.  (See  also  “functional  architecture.”) 


This  term  and  its  use  throughout  the  rest  of  the  model  is  intended  to  encourage 
use  of  proven,  architecture-centric  practices  and  the  recognition  of  “architecture” 
as  a  principal  engineering  artifact. 
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Changes  in  CMMI  Terminology  -  3 


Definition  of  required  functionality  and  quality  attributes 

DEFINITION 

A  characterization  of  required  functionality  and  quality  attributes  obtained  through 
“chunking,”  organizing,  annotating,  structuring,  or  formalizing  the  requirements 
(functional  and  non-functional)  to  facilitate  further  refinement  and  reasoning  about  the 
requirements  as  well  as  (possibly,  initial)  solution  exploration,  definition,  and  evaluation. 

As  technical  solution  processes  progress,  this  characterization  can  be  further  evolved 
into  a  description  of  the  architecture  versus  simply  helping  scope  and  guide  its 
development,  depending  on  the  engineering  processes  used;  requirements 
specification  and  architectural  languages  used;  and  the  tools  and  the  environment  used 
[snip]. 

The  term  “definition  of  required  functionality”  that  appeared  in  VI. 2  has  been 
removed  from  CMMI  because  of  the  implicit  suggestion  that  functionality  be 
addressed  first  or  has  higher  priority.  The  term  has  been  replaced  with  the  one 
above,  which  is  intended  to  help  ensure  a  sufficiently  balanced  and  concurrent 
focus  (functional  and  non-functional)  in  requirements  analysis. 
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Changes  in  CMMI  Terminology  -  4 


“Functional  analysis”  and  “functional  architecture” 

These  terms,  which  appeared  in  earlier  versions  of  CMMI,  are  now  “cul 
de  sacs”  in  the  model. 

The  only  places  these  terms  now  appear  in  CMMI-DEV  VI  .3  outside  of 
the  Glossary  are  in  the  first  note  of  RD  SP  3.2  and  as  an  example 
work  product. 

The  note  in  RD  SP  3.2  contrasts  the  approaches  implied  by  these  terms 
with  “modern  engineering  approaches”  that  encourage  a  more 
balanced  and  concurrent  treatment  of  requirements,  functional  and 
non-functional. 
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Changes  in  CMMI  Terminology  -  5 


Product  line 

DEFINITION 


A  group  of  products  sharing  a  common,  managed  set  of  features  that 
satisfy  specific  needs  of  a  selected  market  or  mission  and  that  are 
developed  from  a  common  set  of  core  assets  in  a  prescribed  wav. 

The  development  or  acquisition  of  products  for  the  product  line  is  based  on 
exploiting  commonality  and  bounding  variation  (i.e.,  restricting  unnecessary 
product  variation)  across  the  group  of  products.  The  managed  set  of  core  assets 
(e.g.,  requirements,  architectures,  components,  tools,  testing  artifacts,  operating 
procedures,  software)  includes  prescriptive  guidance  for  their  use  in  product 
development.  Product  line  operations  involve  interlocking  execution  of  the  broad 
activities  of  core  asset  development,  product  development,  and  management. 

Many  people  use  “product  line”  just  to  mean  the  set  of  products  produced  by  a 
particular  business  unit,  whether  they  are  built  with  shared  assets  or  not.  We  call 
that  collection  a  "portfolio,"  and  reserve  "product  line"  to  have  the  technical 
meaning  given  here. 


Software  Engineering  Institute  Carnegie  Mellon 


Architecture  and  CMMI  VI  .3 

©2011  Carnegie  Mellon  University 


85 


Changes  in  CMMI  Terminology  -  6 


Quality  attribute 

DEFINITION 

A  property  of  a  product  or  service  by  which  its  quality  will  be  judged  by 
relevant  stakeholders.  Quality  attributes  are  characterizable  by  some 
appropriate  measure. 

Quality  attributes  are  non-functional,  such  as  timeliness,  throughput, 
responsiveness,  security,  modifiability,  reliability,  and  usability.  They  have  a 
significant  influence  on  the  architecture. 


This  term  is  now  included  in  the  Glossary  for  the  first  time.  This  term  is  intended 
to  supplant  others  -  especially  those  focusing  on  only  a  few  dimensions  (e.g., 
“performance”)  -  to  encourage  a  broader  view  of  non-functional  requirements. 
The  term  was  refined  through  much  effort,  as  neither  ISO  25030  (SQuaRE)  nor 
the  original  SEI  definitions  were  quite  satisfactory. 
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Changes  in  CMMI  Terminology  -  7 


“Performance”  used  by  itself  can  be  ambiguous 

A  “quality  attribute”  for  CMMI  is  clarity  ©.  One  term  that  has  repeatedly 
caused  problems  in  translations  was  “performance.”  For  VI  .3,  each  use 
of  the  term  was  examined  to  ensure  it  was  unambiguous,  correctly  used, 
and  where  appropriate  the  term  was  qualified: 


-  supplier  performance 

-  project  performance 

-  product  performance 

-  technical  performance 

-  organization’s  performance 

-  cost,  schedule,  performance 

-  performed  process  (CL1) 

-  process  performance 

-  period  of  performance 

-  service  delivery  performance 

-  project  progress  and  performance 

-  fit,  form,  function,  performance 
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Changes  in  CMMI  Terminology  -  8 


Establish  and  maintain 

DEFINITION 

Create,  document,  use,  and  revise  .  .  .  as  necessary  to  ensure  it  remains  they 
remain  useful. 

The  phrase  “establish  and  maintain”  means  more  than  a  combination  of  its  component 
terms;  .  .  .  plays  a  special  role  in  communicating  a  deeper  principle  in  CMMI:  work 
products  that  have  a  central  or  key  role  in  work  group,  project,  and  organizational 
performance  should  be  given  attention  to  ensure  they  are  used  and  useful  in  that  role. 

This  phrase  has  particular  significance  in  CMMI  because  it  often  appears  in  goal  and 
practice  statements  .  .  .  and  should  be  taken  as  shorthand  for  applying  the  principle  to 
whatever  work  product  is  the  object  of  the  phrase. 


The  above  term  appears  in  many  CMMI  practices.  This  term  was  changed  in  VI  .3  to 
emphasize  that  artifacts  that  have  a  long-term  role  need  to  evolve  to  remain  useful. 
Example  from  RD  SP  2.1  note:  “The  modification  of  requirements  due  to  approved 
requirement  changes  is  covered  by  the  “maintain”  aspect  of  this  specific  practice...”  The 
issue  of  how  much  to  document  is  also  addressed,  e.g.,  TS  SP  2.2. 
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Presentation  Outline 


CMMI  VI  .3  -  Context  for  modern  engineering  practices  changes 
Introduction  to  Architecture 

Essential  Architecture  Practices 

Where  Are  the  Architecture-Centric  Practices  in  CMMI  VI  .3? 

Summary 

Questions  and  Answers 
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Summary  &  Conclusions 

The  quality  and  longevity  of  a  software-intensive 
system  is  largely  determined  by  its  architecture. 

Early  identification  of  architectural  risks  saves 
money  and  time. 


There  are  proven  practices  to  help  ensure  that 
suppliers  and  acquirers  can  develop  and  acquire 
systems  that  have  appropriate  architectures. 

CMMI  VI  .3  has  a  new  emphasis  on  architecture. 


The  efficacy  of  the  architecture  has  a  direct 
impact  on  program  or  mission  success,  and 
customer  satisfaction. 
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Objectives 


•  Provide  conference  attendees  with  a  practical 
method  for  estimating  the  project  size  of  ERP 
implementations  that  is  both  easy  to  learn  and 
apply 

•  Compare  the  behavior  of  ERP  implementations  to 
other  business  IT  projects 

■  Size  vs.  Schedule 

■  Size  vs.  Effort 
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The  Intelligence  behind 

Successful  Software  Projects 


Outline 


•  Key  differentiators  between  ERP  implementations 
and  software  development 

•  Sizing  ERP  implementations 

■  Determining  size 

■  RICEF  objects 

■  Configuration  items 

■  Normalizing  to  a  common  metric 

•  Estimating  ERP  implementations 
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Quotations 


“Perfection  is  the  enemy  of  the  possible” 

-  Voltaire  (paraphrased) 

“Precision  is  not  accuracy” 

-  William  Horton 
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Key  Differentiators 


•  Software  projects  create  code 

■  Develop  new  systems 

■  Modify  existing  systems 

■  Are  measured  (sized)  by  the  functionality  they  deliver 
and/or  the  code  they  create 

•  Software  projects  may 

■  Develop  interfaces 

■  Have  hardware,  network,  telecom  components 

■  Convert  data 

■  Have  system  setup  and  configuration 
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Key  Differentiators 


•  ERP  Implementations  have 

■  Significant  system  setup  &  configuration 

■  Hardware,  network,  &  telecom  components 

•  ERP  Implementations  may 

■  Develop  interfaces 

■  Convert  data 

■  Create  additional  functionality 

■  Modify  existing  functionality 
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Determining  Size 


•  Software  project  size  is  not  how  much  it  costs 
nor  how  long  it  takes 

•  Size  measures  the  functionality  a  software 
project  delivers 

•  Parametric  estimation  (SLIM,  COCOMO,  etc.) 
uses  size  as  a  key  input  to  determine  cost  and 
schedule 

■  Lines  of  code,  function  points,  requirements,  use  cases 
are  traditional  size  measures 

•  What  size  measures  capture  the  functionality  of 
an  ERP  implementation? 
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Sizing  ERP  I  mplementations 


•  ERP  Implementation  size:  two  components 

■  Configurations 

■  Customizations 

•  Configurations  include  parameters,  properties, 
rules,  values,  table  setup 

•  Customizations  are  principally  code 

•  Proportions  vary  between  projects 

•  ERP  sizing  must  consider  both 
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Configurations 


•  Estimate  the  number  of  configuration  items  (by 
category  &  complexity) 

■  Best  case,  worst  case,  most  likely  scenarios 

•  Alternatively,  identify  number  of  high  level 
business  processes  that  must  be  configured 

■  SAP  Solution  Composer  is  an  example 

•  Normalize  them  to  a  common  elementary  unit 
(using  gearing  factors) 
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Configuration  Example:  Tables 


•  Average  table  has 

■  3  indices  to  define 

■  20  columns  to  define 

■  20  data  types  (one  per  column) 

•  Average  table  (in  this  example)  requires  43 
elementary  activities  (or  implementation  units)  to 
create 

■  Gearing  factor  of  43 
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Customizations 


•  RICEF  objects:  Reports,  Interfaces,  Conversions, 
Enhancements,  Forms 

•  Estimate  counts  of  each  item  (by  complexity) 

•  Normalize  them  to  a  common  elementary  unit 
(using  gearing  factors) 

•  Add  to  normalized  configuration  items  count  for 
an  estimated  project  size 
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Sample  Gearing  Factor  Table:  Rl  CEF 

Objects 


Component 

Gearing  Factor 

Number 

Size 

Simple  Reports 

100 

10 

1000 

Average  Reports 

200 

5 

1000 

Complex  Reports 

300 

20 

6000 

Simple  Interfaces 

320 

2 

640 

Average  Interfaces 

620 

12 

7440 

Complex  Interfaces 

1520 

1 

1520 

Simple  Conversion 

100 

2 

200 

Average  Conversions 

200 

5 

1000 

Complex  Conversions 

300 

2 

600 

Simple  Enhancements 

100 

2 

200 

Average  Enhancements 

500 

1 

500 

Complex  Enhancements 

1000 

3 

3000 

Simple  Forms 

100 

2 

200 

Average  Forms 

200 

15 

3000 

Complex  Forms 

300 

3 

900 

Total 

27,200 
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But,  Does  it  Work? 


•  Step  1:  Size  completed  ERP  implementations 
using  configuration  items  and  RICEF  objects 

•  Step  2:  Compare  trends  for  Effort,  Schedule, 
Staffing,  and  Productivity  to  trends  for  Business 
IT  projects  (non-ERP) 
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■  —  All  Systems  Avg.  Line  Style  - 1  Sigma  Line  Style 


Schedule 


Blue  lines  are 
trends  for  ERP 
implementations 
Black  lines  for 
Business  IT 
projects 
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Effort 
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Average  Staff 


Average  Staff  vs  Size 


■  —  All  Systems  Avg.  Line  Style  . 1  Sigma  Line  Style 
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Average  Staff 


ERP  implementations 
use  slightly  smaller 
teams  for  most 
projects  although  the 
trends  and  the 
amount  of  variability 
are  very  similar 
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Average  Staff  vs  Size 


All  Systems  —  QSM  2008  Business  ■  A vg .  Line  Style - 1  Sigma  Line  Style 
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Productivity  Parameter 
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ERP  Implementations 
are  more  productive 
than  Business  IT  for 
smaller  projects  but 
lose  their  advantage 
as  size  grows 


All  Systems  - QSM  2008  Business - Avg.  Line  Style - 1  Sigma  Line  Style 
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Conclusions 


•  ERP  Implementations  have  very  similar  behavior 
to  other  Business  IT  projects 

■  Schedule,  effort,  staffing,  productivity 

•  Parametric  estimation  techniques  used  for 
Business  IT  projects  are  applicable  to  ERP 
implementations 

•  ERP  Implementation  size  can  be  effectively 
estimated  using  Configuration  Items  and  RICEF 
Objects 

■  Widely  used  by  U.S.  government  for  estimation  and 
tracking 
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Conclusions 


•  Although  smaller  ERP  implementation  projects 
are  slightly  more  productive  than  traditional 
Business  IT,  the  cost  of  the  package  should  be 
included  in  cost  estimates  if  it  is  being  purchased 

•  While  larger  ERP  implementations  do  not  enjoy 
cost  or  schedule  advantages,  larger  traditional 
Business  IT  projects  have  a  higher  probability  of 
failure,  which  must  also  be  considered  when 
choosing  an  alternative 
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Analogy  "Management  Commitment" 
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WUAT  WOULD  WE 
CALL  rrP 
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Analogy  "Uses  Cases" 


amethodpark 


Context:  Product  Development 


Context:  Process  Development 


Wikipedia: 

•  "A  use  case  in  [...]  system  engineering  is  a  description 
of  a  system's  behaviour  as  it  responds  to  a  request 
that  originates  from  outside  of  that  system.  In  other  words , 
an  use  case  describes  "who"  can  do  "what"  with  the 
system  in  question.  The  use  case  technique  is  used  to 
capture  a  system's  behavioral  requirements  by 
detailing  scenario-driven  threads  through  the 
functional  requirements. " 

•  A  use  case  should: 

■  Describe  what  the  system  shall  do  for  the  actor 
to  achieve  a  particular  goal. 

■  Include  no  implementation-specific  language. 

■  Be  at  the  appropriate  level  of  detail. 


process(es)  (and  output) 
process  input 
role  model 
activities 

process  requirements 
process  tailoring 


relevant  stakeholder 
process  objective(s) 

appropriateness 


•  Analogies  to  talk  about: 

■  What's  an  "actor"  in  a  process  improvement  scenario? 

■  What  are  the  actor's  goals  in  a  process  improvement  scenario? 

■  What  is  "appropriate  level  of  detail"  in  a  process 
improvement  context? 

■  What's  the  "system"  and  the  "system  behaviour"  in  a  process 
improvement  scenario? 
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Goal  Orientation"  and  "Appropriateness" 


□  methodpark 
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"Goals"  and  "Appropriate  Level  of  Detail" 


amethodpark 


Company 


Department 


i 


◄ - ► 


/K 


/K 


/N 


Employees 
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An  Example  on  Company  Level 


amethodpark 


•  Does  the  company's  vision  and  the  project's  vision 
conform  to  each  other? 

•  Scenario  based  business  planning  (->  Use  Cases?) 

•  Typical  questions  (from  the  management  viewpoint): 

■  Where  are  we  (the  organization)  in  5  years? 

■  Who  is  then  our  customer?  What  does  he  need? 

■  Which  products  do  we  have  then? 

■  Which  kind  of  services  will  we  offer? 

■  Who  will  be  our  competitor? 

•  Derived  questions: 

■  Which  organizational  structure  do  we  need  then? 

■  With  whom  will  we  cooperate? 

■  Which  new  technologies  will  we  use? 

■  Which  quality  goals  do  we  have  to  fulfill? 

■  What  kind  of  culture  do  we  need  in  the  organization? 

■  What  kind  of  skills  do  we  need  for  this? 
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Actors  =  Process  Stakeholder  amethodpark 


Actors:  Stakeholder  Analysis 


amethodpark 


•  Document  the  Stakeholders 

■  For  each  Stakeholder 

■  Name 

■  Function  (Role) 

■  Additional  personal  data  /  contact  data 

■  Availability  (time  and  region)  during  the  project 

■  Relevance  of  the  Stakeholder 

■  Knowledge  area  and  scope 

■  Personal  goals  /  interests  regarding  the  project 

•  Stakeholder  Relationship  Management 

■  Convince  Stakeholders  about  the  project's  benefit 
(Motivation!) 

■  Prevents  conflicts 

■  Basis  for  active  Stakeholder  Involvement  during  the  project 
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Stakeholder  Analysis 


amethodpark 


High 


Power 


Low 


L 

Keep 

Satisfied 

Manage 

Closely 

i 

Monitor 

Keep 

(Minimum  Effort) 

Informed 

Low 


Interest 


High 
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More  Details  about  "Use  Cases" 


□  methodpark 


■  User's  point  of  view 

What  is  the  apparent  functionality  of  the 
system? 

■  What  are  the  neighbouring  systems  and 
users? 

■  Where  are  the  system  boundaries? 

■  Use  cases  describe  the  operational  flow  of 
the  system. 

■  What  the  system  should  do,  and  not  how! 

■  Semi-formal,  and  can  be  easily  understood. 


team  members 
process  outcomes 

interfaces  between 
processes  and  tools 

responsibilities 

activities 

Mmmhhh . 

processes  and  methods 
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Use  Cases  for  Use  Cases 


amethodpark 


•  In  Engineering:  "Use  Case  Driven  Development" 


Object  oriented 
analysis 


Project  planning 


Tests  and 
acceptance  criteria 
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Use  Cases:  How  to  Define 


amethodpark 
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Example:  Use  Cases  for  Products 


□  methodpark 
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Example:  Use  Cases  for  Processes 


□  methodpark 


©  Method  Park,  2011  /  NDIA  CMMI  Conference  2011,  Denver  (CO) 


Customer 


15 


Example  Configuration  Management 


amethodpark 


•  Stakeholder,  e.g.: 

■  Controlling 

■  Purchasing 

■  Development  (SW,  EE,  ME) 

■  Manufacturing 

•  Challenges: 

■  Ensure  information  flow 
(The  right  information  to  the  right  place  at  the  right  time) 

■  Different  tools  per  discipline  /  department 

•  Prerequisite: 

■  Know  who  are  the  Stakeholders 

■  Understand  the  Stakeholder's  needs 

■  Understand  their  current  problems: 

Terms,  Processes,  Tooling 
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Summary  Use  Case  Analysis 


amethodpark 


•  Systematic  Use  case  Analysis 

■  Leads  to  an  overview  of  actors 

■  Process  Stakeholders  (team  and  management  roles, 
external  stakeholder) 

■  Places  the  focus  on  the  actors  (stakeholders)  and  therefore 
focus  automatically  on  the  business  needs  (instead  on  the 
"level") 

■  Reduces  complexity  because  the  whole  process  is 
subdivided  in  smaller  use  cases  and  scenarios 

-  Best  practice: 

■  While  discussing  use  cases  ask  the  process  stakeholders  for 
current  issues  with  this  process! 

■  Trace  Use  Cases  and  Process  Issues  to  the  solution! 

■  Supports  to  create  a  tool  map  (overview  of  tools  and 
interfaces  between  them) 
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Use  Cases  for  Use  Cases 


amethodpark 


Use  cases 


Improvement 

Planning 


process  assets 


Use  Cases  and 
Process  Assets 
supports  to  achieve 
completeness 


process  assets 
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Keep  it  Simple  and  Smart! 


amethodpark 
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Lessons  Learned 


amethodpark 


define  CMMI  p 
a  Goal 


r 


•Jefint 

Processes 


<> 


Analyze 
Use  Cases 


LE 


Define 

Processes 


<> 


Maintain 

Traceability 


Review 


U 


Piloting 
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Summary 


amethodpark 


•  Focus  on  Stakeholder 

Goals  &  Needs 

■  Traceability 

Vision  -  Goals  -  Strategy  - 
Use  Cases  -  Problems  -  Solutions 

•  Focus  on  Process  Integration 

■  Mutual  Comprehension 

■  What  does  my  colleague  need  to  be 
able  to  do  the  work? 

■  Build  the  "Big  Picture" 

■  Terms  (Glossary) 

■  Visualize  Process  Interfaces 


Help  the  business,  help  the  people! 
Think  about  what  do  they  really  need! 
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r-n 


amethodpark 


Thank  You  for  your  Attention! 
Questions?  Now  or  later: 


juergen.schmied@methodpark.com 
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Managed  Discovery?  PIIDs? 

How  do  I  decide  which 
SCAMPI  data  collection  techniques 
are  right  for  my  organization? 


Sam  Fogle 

SCAMPI  High-Maturity  Lead  Appraiser 

ACE  Guides,  LLC 


CMMI  Tech  Conf  2011 


Agenda _ 

•  Data  Collection  Approaches 

•  So,  What  Do  I  Build? 

•  Got  Any  Tips? 

•  Problems  We  Have  Met  &  Lessons  We 
Have  Learned 


Data  Collection  Approaches 


Appraisal  Terms  and  Concepts 

•  "To  make  reasonable  judgments  regarding  an 
organization's  implemented  processes  relative 
to  the  appraisal  reference  model,  appraisal 
teams  base  their  judgments  on  the  collection 
of  objective  evidence  for  each  specific  and 
generic  practice  applicable  to  process  area 
goals  within  the  appraisal  scope."* 

•  Objective  evidence  or  "footprints",  left  behind 
after  a  practice  has  been  implemented,  are 
Practice  Implementation  Indicators  (PIIs) 

•  A  mapping  between  model  practices  and  the 
PIIs  is  called  the  Objective  Evidence  Database 


*From  the  SCAMPISM  A,  V  1.3:  Method  Definition  Document 
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Appraisal  Approaches 


1.  Full 

Discovery 


m 


Full  discovery  requires  that  the  appraisal  team 
do  a  search  for  evidence  for  each  practice 

SCAMPIsm  pilots  took  more  than  a  month 

Appraisal  conduct  needed  to  be  streamlined 


Appraisal  Approaches 


2.  Full 

Verification 


Complete 

Mapping 


In  full  verification,  the  organization  provides  a 
mapping  of  evidence  for  each  practice.  The 
appraisal  team  only  searches  when  the  data 
provided  is  not  clear  or  convincing 


This  mapping  could  require  >1K  hrs  to  develop 


A  streamlined  option  for  appraisal  preparation 
was  needed 


Appraisal  Approaches 


3.  Managed 
Discovery 


Mapping 
of  Key 
Artifacts 


Managed  discovery  has  the  organization 
provide  key  artifacts  that  address  the  majority 
of  the  practices  and  the  appraisal  team  asks  for 
additional  evidence  as  required 


The  appraisal  conduct  may  require  additional 
time  but  preparation  is  simplified 


Appraisal  Approaches 


Comparison  of  the  three  approaches: 


Preparation  Conduct  Risk 


Full  Discovery  Low  High  High 

Full  Verification  High  Low  Low 

Managed  Discovery  Moderate  Moderate  Moderate 


SCAMPI  VI. 3  allows  all  of  the  above  or 
anywhere  in  between 

It  is  up  to  the  organization  and  lead  appraiser 
to  plan  a  data  collection  strategy  that  best 
meets  the  organization's  needs 


Do  I  still  need  some  form  of  PHD? 


Probably! 

For  many  appraisals,  the  decision  will  be  made 
that  some  level  of  Objective  Evidence  Database 
or  Mapping  that  relates  CMMI  practices  to  your 
data  will  be  required 


So,  What  Do  I  Build? 


An  Appropriate  Set  of  Artifacts? 

•  Enough  to  convince  the  appraisal  team 
that  a  practice  has  been  implemented* 

•  Multiple  instances  for  items  that  are 
produced  frequently,  e.g.  meeting  minutes 

•  Cover  the  breadth  of  the  practice 


*Talk  to  your  Lead  Appraiser 


Who  Should  Build  Your  Mapping? 

What  kind  of  knowledge  is  needed? 

•  Understanding  of  the  model  practices 

•  Understanding  of  the  appraisal  method 

•  Understanding  of  how  the  project's  data  is 
organized 

Do  you  have  one  person  with 
If  not,  then  you  need  a  team 

•  Process  person? 

•  Project  representative? 

•  Consultant? 
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all  3? 


Mapping  Contents _ 

•  CMMI  ®  practice  name 

•  Artifact  name 

•  Artifact  location 

•  Basic  Unit  (Project)  or  Support  Group 

•  Comments  (may  explain  relevance  of 
data) 


Mapping  Format _ 

•  Does  your  Lead  Appraiser  have  a  tool 
that  they  recommend? 

•  Will  you  use  the  data  for  internal 
purposes? 

•  How  easy  is  it  to  insert  or  retrieve  data? 

•  How  easy  is  it  to  correct/update  the 
data? 

•  Will  you  map  practices  to  artifacts  or 
artifacts  to  practices? 


Cost  &  Timing _ 

•  Objective  Evidence  Database  development 
can  be  a  major  driver  of  total  appraisal  cost 

•  The  highest  quality  will  come  from 
developing  the  Objective  Evidence  Database 
in  several  iterations,  so  start  early 

•  If  it  is  possible  to  use  the  same  sampling  in  a 
series  of  appraisals  (SCAMPI  B,  then  SCAMPI 
A),  then  going  through  the  SCAMPI  B  will 
identify  many  of  the  Objective  Evidence 
Database  issues  and  allow  corrections  to  the 
data  collection  process  prior  to  the  SCAMPI  A 
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Validation  _ 

•  Involve  team  members  in  quality  checks 

•  Don't  wait  for  Readiness  Review  to  check  the 
quality  of  the  Objective  Evidence  Database 

•  Evolve  mapping  over  a  series  of  appraisals 

•  Effort  devoted  to  the  Objective  Evidence 
Database  (including  quality  checks)  should  be 
proportional  to  the  importance  of  achieving 
the  ratings 

Remember:  having  an  inaccurate  mapping  does  not  just 
make  it  harder  to  find  the  correct  data,  it  may  convince 
the  appraisal  team  that  appropriate  data  does  NOT  exist 


Got  Any  Tips? 


Potential  Work  Saver _ 

Is  the  standard  process  detailed  enough  so 
that  it  will  be  fairly  consistent  between 
implementations  where  a  specific  type  of 
data  will  be  found? 

If  so,  then  provide  the  units/projects  with 
a  mapping  that  already  tells  them  where  to 
find  the  evidence 

e.g. 

PP  SP  2.2  Identify  Project  Risks 

see  section  3.4  of  the  Risk  Management  Plan 


Use  Directories 

When  the  evidence  you  want  to  include  is  a 
frequently  generated  item  (e.g.  monthly 
meeting  minutes  or  peer  review  reports) 
linking  to  the  directory  where  the  items  are 
stored  will  provide  advantages 

•  The  link  will  not  go  stale  -  new  items  will 
continue  to  be  populated  into  the  directory 
so  fresh  evidence  is  available 

•  The  appraisal  team  is  free  to  sample  from 
the  set,  thus  aiding  confidence  in 
institutionalization 


Problems  We  Have  Met  & 
Lessons  We  Have  Learned 


Problems 


•  What  sort  of  problems  have  you 
encountered  in  trying  to  plan  for  or 
develop  your  mappings? 


•  Have  others  encountered  this  same 
problem?  And  if  so,  were  you  able  to 
solve  it? 


Lessons  Learned 


•  What  other  lessons  have  you  learned 
related  to  Objective  Evidence  Databases? 


Mapping  Preparation _ 

If  done  poorly 

•  Can  consume  vast  resources  to  prepare 

•  Will  reflect  a  poor  understanding  of  what  is  needed 

•  Will  cause  appraisal  to  proceed  very  slowly 

•  Can  confuse  the  true  state  of  the  practice 

If  done  well 

•  Will  require  limited  restarts  or  rework 

•  Accurately  reflects  the  work  done  in  the  organization 

•  Provides  an  efficient  means  for  an  appraisal  team  to 
find  appropriate  evidence 

•  Identifies  appraisal  risks  by  uncovering  holes  in 
implementation 


Questions ! 


Contact  Info _ 

•  Sam  Fogle,  Chief  Guide,  SEI-Certified 
SCAMPI  High-Maturity  Lead  Appraiser 

•  ACE  Guides,  LLC 

•  www.ACEguides.com 

•  sam@ACEquides.com 

•  240-308-0767 


®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SMSCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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Agenda 


Raytheon 

Intelligence  and 
Information  Systems 


■  CMMI  History 

■  Operations  Goals 

■  Understanding  CMMI-DEV  vl  .3  and  CMMI-SVC  vl  .3 

■  What  Does  SEI  Recommend? 

■  Evaluation  Approach 

■  Evaluation  Analysis 

-  Costs  and  Effort 

-  Evaluation  Parameters 

-  Decision  Analysis  and  Resolution  (DAR) 

■  Recommendation 

■  Current  Activities  and  Plans 

■  Conclusion 
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Raytheon 

Intelligence  and 
Information  Systems 


CMMI  Background  and  Highlights 

□  1998  SDSIO  Contract  Awarded  to  Raytheon  by  NASA’s  Jet  Propulsion 
Laboratory 

-  2004  Achieved  CMMI  Maturity  L3  for  Services  (CMMi-Dev,  version  i.i) 

•  Innovative  approach  of  to  applying  CMMI-DEV  to  Services  Organization 

•  Received  patent  (05E095) 

-  2007  Achieved  CMMI  Maturity  L3  for  Services  (CMMi-Dev,  version  1.2) 

-  2007  Achieved  CMMI  Engineering  Capability  for  Raytheon  Web 
Solutions  (RWS)  (CMMI-Dev,  Version  1.2) 

•  Pasadena  Innovative  approach  was  published  as  a  use  case  in  the  CMMI- 
DEV  Version  1 .2  book 

□2008  Won  the  re-compete,  DSIO  Contract  Awarded. 

•  CMMI  was  a  key  discriminator 

•  CMMI  is  a  DSIO  contract  requirement 

-  2010  Achieved  CMMI  L3  Maturity  L3  for  Services  (cmmi-dev,  version  1.2) 

Continuous  process  improvement  culture 
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Operations  Goals 


Raytheon 

Intelligence  and 
Information  Systems 


□  Overview 

-  CMMI-DEV  vl  .1  was  implement  for  2004  certification 

-  CMMI-DEV  vl  .2  was  implemented  for  2007  &  2010 

-  CMMI-DEV  vl. 3  and  CMMI_SVC  vl.3  released  late  2010 

□  Raytheon  Pasadena  Operations  Goals 

-  Maintain  customer  focused  approach  to  processes 

-  Maintain  Raytheon  Pasadena  CMMI  certification 

-  Re-evaluate  CMMI  constellation  selection 

-  Identify,  document  and  make  a  recommendation  to  leadership  on 
which  CMMI  constellation  is  best  suited  for  Raytheon  Pasadena 


CMMI  is  aligned  with  Raytheon  Pasadena  goals 
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Understanding 
CMMI-DEV  and  CMMI-SVC 


Raytheon 

Intelligence  and 
Information  Systems 


□CMMI-DEV  vl  .3 

•  Has  a  total  of  1 8  Process  Areas  (PAs) 

•  From  which  17  PA  directly  apply  to  Pasadena  Operations 

•  The  Supplier  Agreements  Management  (SAM)  PA  is  not  implemented 

•  For  Maturity  Level  3  12  out  of  the  18  PA  are  the  same  for  CMMI-DEV  and  CMMI-SVC 

•  For  Maturity  Level  3  5  PAs  are  unique  to  CMMI-DEV 


□CMMI-SVC  vl.3 

•  Has  a  total  of  1 9  PA 

•  Could  primarily  re-use  12  PAs 
from  the  existing 
implementation  of  CMMI-DEV 


Could  we  leverage  the  overlap  between  CMMI-DEV  and  CMMI-SVC? 
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Raytheon 

CMMI-SVC  mapped  to  Existing  Process 


SVC  Practice  Area 


Coverage 

Rating  | Comments 


Service  Delivery  (SD) 


Incident  Resolution  and  Prevention  (IRP) 


Service  System  Design  (SSD)  (optional) 


Service  System  Transition  (SST) 


Strategic  Service  Management  (STSM) 


Capacity  and  Availability  Management  (CAM) 


Service  Continuity  Management  (SCON) 


Covered  well  with  existing  processes.  No  gaps.  Service 
Agreement  is  the  SWO  (Subcontract  Work  order); 
Service  system  is  the  combination  of  the  5  service 
components  and  is  defined  by  the  WCP;  Service  System 
component  is  one  of  the  golden  5  service  provision 
requirements;  Service  System  Delivery  is  the  DSIO 
| Contract  Tool;  Service  Request  is  the  SOW; 


I  Little  or  No  coverage  with  existing  processes.  Many 
Igaps.  An  incident  is  and  indication  of  a  problem  or 
1  interference  with  service  delivery.  Typical  instantiation  of 
■this  would  be  help  desk  with  tickets.  A  possible 
I  approach  would  be  to  use  the  DSIO  contract  tool  to 
1  record  incidents  from  the  customer. 


Covered  well  with  existing  processes.  No  gaps.  Well- 
covered  with  existing  Engineering  PAs;  however 
opportunity  exists  to  minimize  engineering  processes 
with  SVC. 


Covered  well  with  existing  processes.  No  gaps.  A 
Service  system  is  represented  and  defined  by  the  WCP; 
Service  Systems  are  all  of  the  WCPs  under  the  DSIO 
contract.  A  request  for  a  change  (MOD)  in  service  would 
be  a  service  request;  so  a  service  transition  would  occur 
as  a  result. 


Some  coverage  with  existing  processes.  Some  Gaps. 
Unclear  as  to  what  the  standard  services  are  and  how 
they  are  presented  to  current  and  potential  customers. 


Some  coverage  with  existing  processes.  Some  Gaps. 
Personnel  and  Facilities  planning  exist;  would  need  more 
emphasis  of  coordination  across  all  services,  as  well  as 
modelling,  measurement,  and  analysis  to  make  sure  that 
all  necessary  resources  for  service  delivery  are  at  the 
right  levels  and  available  when  needed. 


I  Little  or  No  coverage  with  existing  processes.  Many 
Igaps.  No  continuity  planning  in  place. 
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What  does  SEI  recommend? 


Raytheon 

Intelligence  and 
Information  Systems 


□Excerpts  from  CMMI  for  Services,  vl  .3  technical  report 

“Organizations  interested  in  evaluating  and  improving  their  processes  to  develop  systems 
for  delivering  services  can  use  the  CMMI-DEV  model.  This  approach  is  especially 
recommended  for  organizations  that  are  already,  usincj  CMMI-DEV  or  that  must  develop 

and_  rnaintain  complex  systems  for  delivering,  services.  However,  the  CMMI-SVC  model 
provides  an  alternative,  streamlined  approach  to  evaluating  and  improving  the 
development  of  service  systems  that  can  be  more  appropriate  in  certain  contexts.” 

“Service  provider  organizations  can  also  choose  to  use  the  CMMI-DEV  model  as  the  basis 
for  improving  and  appraising  their  service  system  development  processes.  This  use  of 
the  CMMI-DEV  model  is  preferred  for  organizations  that  are  already  experience d  with 

CMMI-DEV  and_  for  organizations  that  develop  large-scale ,  complex  service  systems.  ” 

” Even  organizations  that  use  the  CMMTDEV  model  for  service  system  development  may 
wish  to  refer  to  the  Service  System  Development  pmcess  area  for  helpful  guidance  on 

applying  development  practices  to  service  system  components  such  as  people, 
processes,  and  consumables.  ” 


SEI:  Considers  the  organization’s  needs  and  context 


11/17/11 
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Evaluation  Approach 


Raytheon 

Intelligence  and 
Information  Systems 


□  Would  CMMI-SVC  work  in  our  environment? 

-  We  were  able  to  map  our  business  model  to  the  CMMI-SVC  model 

-  However,  typical  industry  service  agreements  are  repetitive;  the 
same  service  is  offered  to  multiple  customers. 

-  Our  customer  service  agreements/sub-contracts  are  unique 

-  The  process  to  instantiate  a  new  sub-contract  is  the  repetitive 
component 


Addressing  our  customer  needs  and  requirements 


11/17/11 
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Analysis  -  Effort  &  Costs 


Raytheon 

Intelligence  and 
Information  Systems 


□What  is  the  impact  of  changing  to  CMMI-SVC  in  Pasadena? 

-  Is  what  we’re  doing  good  enough? 

-  Are  we  meeting  our  customer  requirements  and  is  our  customer  happy? 

-  How  are  our  operations  impacted  with  the  implementation  of  CMMI-SVC? 

□CMMI-SVC  Implementation  Effort 

-  Cost  associated  with  implementation  of  7  new  PAs 

•  5  PAs  effort  to  implement  is  “easy”  to  “moderate” 

•  2  PAs:  Incident  Resolution  and  Prevention  (IRD),  and  Service  Continuity 
Management  (SCON)  are  high  impact  to  our  current  operations  and  complex  to 
implement 

-  Cost  associated  with  creating  the  support  infrastructure 

-  CMMI-SVC  has  2  more  PAs  than  CMMI-DEV  this  may  increase 
evidence  collection  and  SCAMPI  effort 


Add  value  to  our  customers  -  no  additional  costs 
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Analysis  -  Solution  Alternatives 


Raytheon 

Intelligence  and 
Information  Systems 


□  How  does  Raytheon  Pasadena  want  to  be  recognized? 

-  As  a  product  developer? 

-  As  a  service  provider? 

-  As  both  a  product  developer  and  a  service  provider? 

-  As  a  combination  or  mixture  of  the  two? 

-  As  not  interested  in  CMMI? 


Maintain  positive  customer  relationships  and  reputation 


11/17/11 
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Analysis  -  Evaluation  Parameters 


Raytheon 

Intelligence  and 
Information  Systems 


□  Identified  Key  Evaluation  Parameters 

-  Selected  alternatives 

•  Continue  using  CMMI-DEV 

•  Implement  CMMI-SVC 

•  Continue  using  CMMI-DEV  and  implement  CMMI-SVC 

•  Continue  using  CMMI-DEV  and  implement  CMMI-SVC  capability 

•  No  CMMI  re-certification 

-  Defined  evaluation  criteria 

•  Impact  to  Raytheon  Pasadena 

•  Reputation 

•  Costs 

•  Benefits  to  the  Organization 

•  Customer  Satisfaction 

-  Assigned  weights  to  evaluation  criteria 

•  Sane  weights  were  assigned  to  identified  evaluation  criteria  based  on  identified  goals 


Essential  characteristics  of  Raytheon  Pasadena  CMMI  efforts 


11/17/11 
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Analysis  -  Raytheon 

Decision,  Analysis,  and  Resolution  (DAR)  S2T“i. 


□  Evaluate  Alternative  Solutions 

-  Assign  values  (1 ,3  or  9)  to  identified  solutions  based  on  analysis  and 
experience 

Decision  Analysis  and  Resolution  Raytheon  Proprietary 


Raytheon 


'WM 

CMMI-DEV  vs  CMM-SVC 

criteria 

T 

NOTE- This  summi 
100,  when  the  nun 
into  the  yellow  ce 

Criteria 

Weight 

Less  Impact  to 
Ops 

Reputation 

(Internal) 

Cost 

Increased 
Benefit  to  MOS 

Increased 

Customer 

Satisfaction 

1 

100 

20 

20 

20 

20 

20 

Alternatives 

1  3  9 

1  3  9 

1  3  9 

1  3  9 

1  3  9 

Ranking 

CMMI-DEV 

9 

9 

3 

3 

i 

s  541  J  , 

180 

180 

60 

60 

60  * 

CMMI-SVC 

3 

3 

3 

3 

3 

300  v 

60 

60 

60 

60 

60 

CMMI-DEV  and  CMMI-SVC 

3 

9 

1 

9 

3 

500 

60 

180 

20 

180 

60 

II 

CMMI-SVC  4  CMMI-DEV  Capability  in 

3 

3 

1 

3 

3 

260 

Engineering 

60 

60 

20 

60 

60 

VI 

CMMI-DEV  +  CMMI-SVC  Capability 

3 

9 

1 

3 

3 

380 

60 

180 

20 

60 

60 

IV 

No  CMMI  re-certification 

9 

1 

9 

1 

3 

460 

180 

20 

180 

20 

60 

III 
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Recommendation 


Raytheon 

Intelligence  and 
Information  Systems 


□  Based  on  DAR  -  Recommended  to  continue  using  CMMI-DEV  for 
Raytheon  Pasadena 

-  Less  impact  to  current  operations 

-  No  increase  costs 

-  CMMI-DEV  has  a  perceived  “reputation”  value 

-  CMMI-DEV  has  potential  to  increase  benefits  to  parent  organization  to  attract 
new  business  (Specialized  Services  Development) 

□  A  close  second  -  Continue  using  CMMI-DEV  and  add  CMMI-SVC 
PA 

-  This  would  mean  implementing  and  assessing  7  new  Process  Areas.  This 
would  exceed  existing  budgets. 

-  Recommend  to  see  about  implementing  some  services  best  practices  from 
CMMI-SVC  such  as  Incident  Resolution  and  Prevention  (IRP),  Capacity  and 
Availability  Management  (CAM)  and  Service  Continuity  (SCON)  to  provide 
higher  value  to  the  organization. 


“In  God  We  Trust,  all  others  bring  data” 
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Raytheon  Pasadena 

Current  Activities  and  Future  Plans 


Raytheon 

Intelligence  and 
Information  Systems 


□2011  CMMI  Activities 

•  White  paper  on  CMMI-DEV  vl  .3  versus  CMMI-SVC  vl  .3  evaluation  and 
recommendation 

•  Recommendation  is  to  continue  using  CMMI-DEV 

•  Transitioned  from  CMMI-DEV  vl  .3  from  vl  .2 

•  SCAMPI  C  scheduled  for  December  12-14,  2011 

□  2012 

•  Continue  process  improvements  and  process 
simplification  efforts 

•  Consider  implementing  identified  CMMI-SVC  PAs 

•  Conduct  SCAMPI  B2 

□  2013 

•  Conduct  SCAMPI  A:  Achieve  CMMI-DEV  Maturity  L3  (Services  adaptation) 


Customer  Success  Is  Our  Mission! 


11/17/11 
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Raytheon 

Intelligence  and 
Information  Systems 


Questions 


11/21/2011 
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Raytheon 

Customer  Success  Is  Our  Mission 


One  if  by  Land,  Two  if  by  Sea... 

You  are  a  CMMI-DEV  appraisal  expert.  What  do 
you  do  if  your  org  wants  to  do  a  CMMI-SVC 

appraisal? 

Debra  Smith 
Kerry  Trujillo 


November  14-17,  2011 


Copyright©  201 1  Raytheon  Company.  All  rights  reserved. 
Customer  Success  Is  Our  Mission  is  a  trademark  of  Raytheon  Company. 


Agenda 


Raytheon 


■  Overview 

■  Beginning  of  the  Story 

■  Raytheon  IDS  Organization  and  Service 

■  Process/Tools/Roles 

■  Summary 

■  End  of  the  Story 

■  Questions 

■  Presenters’  Biographies 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


Overview 


Raytheon 


How  do  you  collect  evidence  the  first  time  an  organization  has  a  CMMI®  for 
Services  appraisal  after  only  being  appraised  against  CMMI  for 
Development? 

How  do  you  know  what  evidence  to  gather  when  you’ve  done  CMMI  for 
Development  appraisals  for  years? 

Raytheon  Integrated  Defense  Systems  (IDS)  worked  with  Cooliemon  LLC, 
an  external  CMMI  appraisal  lead,  to  focus  on  the  process,  tools,  and 
personnel  roles  that  would  create  a  successful  appraisal. 
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Raytheon 


Beginning  of  the  Story 


■  The  time:  April  1775 

■  The  Place:  Boston,  Massachusetts 

■  The  Story: 

Paul  Revere  had  many  occupations;  silversmith, 
dentist,  Lieutenant  in  the  Massachusetts  militia, 
and  Son  of  Liberty. 

As  a  successful  businessman  he  planned  his 
activities,  worked  with  suppliers,  and  dealt  with 
product  defects  and  customer  complaints. 

He  also  became  a  courier  for  the  Boston 
Committee  of  Correspondence  and  the 
Massachusetts  Committee  of  Safety  .  He 
provided  news  on  British  movements  to  people  in 
towns  from  Boston  to  New  York. 
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Raytheon  -  Integrated  Defense  Systems  (IDS) 


IDS  Mission  Centers 
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Raytheon 

Integrated  Defense  Systems 


Integrated  Defen se  Systems  is 
Raytheon's  leader  in  Global 
Cap  Julies  integration  through 
connected  people,  performance, 
relationships  and  solutions. 
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IDS  Life-cycle  Sustainment  Service 


Raytheon 


Curriculum  Development  IPT/CPT  Leadership 
and  Training 


RAM/FRACA 
Depot  Operations 

Maintenance/Disposal 

Supply  &  Field  Support 


Systems  Integrity 

Safety 


Warranty  Management 


Life-cycle  Sustainment 


Parts  Availability 
Provisioning 
Logistics/Spares 
Return  &  Repair 


HSI  CAIV/LCC/TOC 


Tech  Manuals 
Risk  and  Opportunity  Mgmt 


Providing  a  solution  that  ensures  the  system’s 
availability/sustainability/supportability,  as  documented  in  a  contract.  It 
consists  of  multiple  functions  that  are  required  to  fulfill  the  contract. 


Note:  Acronyms  spelled  out  in  the  slide  notes 
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IDS  CMMI  -  SVC  Timeline 


Raytheon 


- ^ - 

w 

V 

l^\ 

First 

Deploy 

O  /OAAA 

CMMI 

CMMI 

CMMI- 

CMMI  SVC 

Class  C 

Class  B 

SVC 

Workshop 

8/2009 

Appraisal 

Appraisal 

SCAMPI 

2/2009 

2/2010 

7/2010 

Level  3 

11/2010 

CMMI  Class  C  Appraisal  -  Looked  at  evidence  and  conducted  interviews.  Lots  of  discovery. 
CMMI  Class  B  Appraisal  -  Looked  at  evidence  and  conducted  interviews.  Very  little  discovery. 
SCAMPI  Appraisal  -  Official  appraisal  that  results  in  a  Maturity  Level  rating. 
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Process  /  Tools  /  Roles 
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CMMI-SVC  and  CMMI-DEV  - 1 


Raytheon 


Today  Reuse 


Tomorrow 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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CMMI-SVC  and  CMMI-DEV  -  2 


Raytheon 


CMMI-DEV 


CMMI  -  SVC 


CMMI  for  Development 


CMMI  for  Services 


Product  Integration  (PI) 
Requirements 
Development  (RD) 
Technical  Solutions 
(TS) 

Validation  (VAL) 
Verification  (VER) 


Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 

Decision  Analysis  and  Resolution  (DAR) 
Integrated  Project/Work  Management 
(IPM/IWM) 

Measurement  and  Analysis  (MA) 
Organizational  Process  Definition  (OPD) 
Organizational  Process  Focus  (OPF) 
Organizational  Performance  Management 
(OPM) 

Organizational  Process  Performance  (OPP) 
Organizational  Training  (OT) 

Process  and  Product  Quality  Assurance 
(PPQA) 

Project/Work  Monitor  and  Control  (PMC/WMC) 
Project/Work  Planning  (PP/WP) 

Quantitative  Project/Work  Management 
(QPM/QWM) 

Requirements  Management  (REQM) 

Risk  Management  (RISKM) 

Supplier  Agreement  Management  (SAM) 


Capacity  and 
Availability 
Management  (CAM) 
Incident  Resolution 
and  Prevention 
(IRP) 

Service  Continuity 
(SCON) 

Service  Delivery 
(SD) 

Service  System 
Development  (SSD) 
Service  System 
Transition  (SST) 
Strategic  Service 
Management 
(STSM) 
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Leverage  from  the  Common 
Process  Area  Evidence 


Raytheon 


Product  integration  (PI) 

Requirements 

Development  (RD) 
Technical  Solutions 
<TS) 

Validation  (VAL) 
Verification  (VER) 


Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 
Decision  Analysis  and  Resolution  (DAR) 
Integrated  ProjectM/orkAlanagement 
(IPM/IWM) 


ood) 


Measurement 
1  Organizational. 

Organizat 
Organizaf 
(OPM) 

Organi; 

Organ!" 

Proces 
(PPr 
Pro) 

Projei 
Quam 

(QPM/CvvM) 

Requirements 
Risk  ManagemelJffdSKM' 

Supplier  Agreement  Management  (SAM) 


Control  (PMC/WMC) 
•PWP) 

>rk  Management 


Capacity  and 

Availability 
Management  (CAM) 
Incident  Resolution 
and  Prevention 

(IRP) 

Service  Continuity 

(SCON) 

Service  Delivery 
(SD) 

Service  System 
Development  (SSD) 
Service  System 
Transition  (SST) 
Strategic  Service 
Management 
(STSM) 


Configuration  Management  (CM) 

•  CM  Plan  is  program  level 

Risk  Management 

•  Risk  Management  Plan  is  program  level 

Work  Monitor  Control 

•  Overall  Program  Schedule 

Measurement  and  Analysis 

•  Measurement  Analysis  Plan 

•  Program  Monitor  Package 
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CMMI  for  Services  for  IDS 


Raytheon 


Customer 
Negotiations  and 
Contract  Award 


Bidding 


Design  influence  and  preparation  to 
ensure  Operational  Availability, 
Sustainability,  Maintainability,  and  Mission 

Assurance 


CMMI  for 
Development 


Business 

Development 


O) 


CD  O 

e  a 


CD 

C  -2  ( J) 

»  g-o 

LLI  Q  0 


<- 

<- 


A  A 


v  v 


o 

o 

"D 

o 


c 

o 

o 

a) 

x 

LU 

"O 

o 


Q- 


Product  Delive 
and  Service 
Initiation 

Fielded  System 
Performance 


Failure  Review 
Board 

Corrective  Action 
Requests 


Concept 


Start-up 


Production  Readiness 


Page  12 


IDS  Organization  Example 


Raytheon 


IBT  =  Integrated  Business  Team  is  the  portion  of  the  IDS  organization 
representing  a  grouping  of  programs  that  provides  products  and  services. 
CBT  =  Cross  Business  Team  is  the  portion  of  the  IDS  organization  that 
provides  people,  processes,  tools  and  technologies  across  IDS  Integrated 
Business  Teams  (IBT). 
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Program  Organization  Example 


IBT 


CPT  =  Cross  Product  Team  is  a  functional  team 
responsible  for  establishing  the  process  and  providing 
resources  to  support  the  IPT. 


IPT  =  Integrated  Product  Team  is  a  multi-functional  team 
responsible  for  delivering  a  product  or  service. 


X  =  Represented 
in  Appraisal 
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Appraisal  Preparation  Roles 


Raytheon 


Appraisal  Team  Representative/Member 

•  Provided  CMMI-SVC  model  interpretation 

SME  for  each  function 

•  Created  “long”  evidence  list 

•  Provided  experiential  knowledge 

•  Did  not  need  to  know  the  model 

Point  of  contact  for  each  program 

•  Deployed  updated  process 

•  Collected  evidence  as  the  program  executed 

•  Statused  readiness 
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Keys  (Reins)  to  the  successful  appraisal  - 1 


Raytheon 


Alignment  between  Appraisal  Team  and  the  Programs 
•Appraisal  Team  overview  and  discussion  meetings 
•  War  Room 

•Appraisal  Team  representative  provided  live 
feedback  to  the  evidence  collectors 

•  Questions  were  answered  quickly 

•  Review  and  validation  of  evidence  was 
immediate 
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Keys  (Reins)  to  the  successful  appraisal  -  2 


Raytheon 


Evidence  collection  tool 

•  Common  repository 

•Leverage  from  the  evidence  for  the  core  processes 

•  Plans,  Measurement  Reviews,  Schedules,  ... 

•  Initial  list  provided  from  Execution  SMEs 

•  They  own  or  use  the  processes  everyday  -  use 
their  knowledge 

•  This  became  a  long  list . 
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Evidence  file  example  -1 


Raytheon 


Generic  Name 

Process 

Area 

W 

Practice 

W 

Description  of  Evidence 

Link 

Comment 

Updated 

CPT  Meeting 
minutes 

IWM 

SP  1.5 

D:  IPT/CPT  Meeting 
Logistics 

httDs://teamexample. Repository. 

com 

CPT  Meeting 
minutes 

WMC 

SP  1.1 

D:  CPT  Meeting  Logistics 

https  ://teamexam  pie.  Repository. 

com 

Supportability  merged  into 
Logistics  CPT  after  reorg  in 
2009-  hence  no  meeting 

CPT  Meeting 
minutesxl 

WMC 

SP  1.5 

PMT  meeting  minutes 

https://teamexample.  Repository. 

com 

PMT  Meeting  minutes  folder - 
see  3/23/10  file 

CPT  Meeting 
minutesx2 

WMC 

SP  1.5 

Team  Handbook 

https://teamexample. Repository. 

com 

See  page  14  (Fig  7)  for  names 
of  each  roles 

CPT  Meeting 
minutes 

WMC 

SP  1.2 

D:  -  CPT  Meeting 

Logistics 

https://teamexample. Repository. 

com 

CPT  Meeting 
minutes 

SD 

SP  2.2 

D:  CPT  Meeting  Logistics 

https://teamexample. Repository. 

com 

Folder  includes  examples  of 
CPT  meeting  minutes 

IPT  Meeting 
minutes 

DAR 

GP  2.8 

D:  IPT  Meeting  Minutes 
where  trade  study 
discussed. 

https://teamexample. Repository. 

com 

Updated  6/22  with 
Comm  Pwr  mm 

IPT  Meeting 
minutes 

DAR 

GP  2.7 

D:  IPT  Meeting  Minutes 
where  trade  study 
discussed. 

https://teamexample. Repository. 

com 

IPT  Meeting 
minutesxl 

SD 

SP  2.2 

D:  IPT  Meetings 

https  ://teamexam  pie.  Repository. 

com 

IPT  Meeting  minutes; 

Operations  is  an  IPT  within 
program  structure;  Fire  Unit 
meetings  is  part  of  the  test 
Radars  IPT  that  is  responsible 
for  maintaining  radar 

Done -5/1 7/10 

Meeting  minutes  were  used  in  31  instantiations 
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Evidence  file  example  -  2 


Generic  Name 

,/ 

Process 

Area 

Practice 

Description  of  Evidence 

Link 

Comment 

Updated 

Management 

Plan 

WP 

SP  1.4 

D:Management  Plan 
Section  4.1 .1 

https://teamexample. Repository. 

com 

See  WLPMP  in  PDM 

Management 

Plan 

WP 

SP  2.7 

D:  Management  Plan 

https://teamexample.  Repository. 

com 

Management 

Plan 

WP 

SP  3.1 

D:  Management  Plan 

https://teamexample.  Repository. 

com 

Management 

Plan 

WP 

GP  2.4 

D:  Management  Plan 
Section  6.2.3  (Roles  and 
Responsibilities) 

https  ://teamexam  pie.  Repository. 

com 

Management 

Plan 

CAM 

GP  2.2 

D:  Mgmt  Plan,  CAM 
planning  sect  4, 

Execution,  Sect  5 

https://teamexample. Repository. 

com 

Management 

Plan 

CM 

GP  2.2 

D:  Management  Plan 

https://teamexample.  Repository. 

com 

Management 

Plan 

DAR 

GP  2.4 

D:  Management  Plan  - 
Roles  &  Resp.  Table  4-1  - 
Program  Management 
Team  does  program 
management  iaw  with 
section  4 

https://teamexample. Repository. 

com 

Management 

Plan 

DAR 

GP  2.2 

D:  Management  Plan 
section  4.12 

https  ://teamexam  pie.  Repository. 

com 

Updated  6/7 

Management 

Plan 

IWM 

SP  1.4 

D:  Management  Plan 

https  ://teamexam  pie.  Repository. 

com 

The  program  plan  was  used  in  42  instantiations 
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Keys  (Reins)  to  the  successful  appraisal  -  3 


Raytheon 


After  a  C  appraisal  the  evidence  list  was  abridged  to 
the  important  few 

•  5%  reduction  in  total  evidence  count  after  the  C 
appraisal 

•  8%  reduction  in  total  evidence  count  after  the  B 
appraisal 

•  Evidence  collection  and  validating  activities 
prior  to  event  significantly  improved,  with  less 
discovery  between  the  C  and  the  B  appraisals 

•  85%  reduction  in  number  of  Document 
Requests  written  during  the  B  appraisal 
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Summary 


Raytheon 


Organizational  support 

•  Knowledgeable  Appraisal  Lead 

•  Service  SMEs 

•  Appraisal  SMEs 

•  Program  points  of  contact 

Tie  your  “Service”  to  your  business  goals  and  objectives. 

•  Collect  evidence  that  is  already  part  of  how  you  do  business. 

•  Have  CMMI  model  SMEs  and  service  SMEs  collaborate. 

Reduction  in  evidence  collection  churn 

•  Tool  that  identifies  the  top  pieces  of  evidence  to  collect  with  mapping  to 
the  model 

•  Leverage  from  the  CMMI-DEV  evidence  list 

COLLABORATION 
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End  of  the  Story 


Paul  Revere  was  successful  in  delivering  his  message  that 
the  Regulars  were  coming.  We  have  evidence  that  he 
planned  for  his  service,  handled  resources  and  incidents, 
and  executed  to  the  plan. 
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Questions 


Raytheon 
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Presenters  Biographies 

Kerry  Trujillo 

Raytheon,  Affordability  Engineering  Section  Manager 

■  Kerry  has  20  years  of  experience  in  military,  commercial,  and  government 
contractor  environments.  He  has  also  held  roles  as  individual  contributor, 
process  engineer,  line  supervisor,  automation  engineer,  project  lead, 
process  group  lead,  and  Naval  Officer.  He  has  utilized  the  CMMI  for 
Services  model  in  a  business  that  maintains  multiple  sustainment  types  of 
contracts  (e.g.  Repair  and  Return,  Performance  Based  Logistics,  Depot 
Support,  and  Field  Operations). 

Debra  Smith 

Raytheon,  Process  Group  Lead 

■  Debra  has  worked  more  than  15  years  in  commercial  and  government 

contractor  environments.  Her  roles  have  included  research  biologist, 
software  engineer,  process  engineer,  IPT  Lead,  six  sigma  black  belt,  line 
supervisor,  and  project  manager.  Debra  is  currently  working  for  a  large 
aerospace  company  on  business  improvement  initiatives  that  involve  both 
CMMI  for  Development  and  CMMI  for  Services.  , 
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Acronyms 

CAIV  =  Cost  As  an  Independent  Variable 
CBT  =  Cross  Business  Team 
CMMI  =  Capability  Maturity  Model  Integration 
CPT  =  Cross  Product  Team 

FRACA  =  Failure  Reporting  Analysis  and  Corrective  Action 

HW  =  Hardware  Engineering 

HSI  =  Human  Systems  Integration 

IBT  =  Integrated  Business  Team 

IPT  =  Integrated  Product  Team 

LCC  =  Life  Cycle  Cost 

RAM  =  Reliability  Availability  and  Maintainability 

SCAMPI  =  Standard  CMMI  Appraisal  Method  for  Process  Improvement 

Sys  =  Systems  Engineering 

SW  =  Software  Engineering 

TOC  =  Total  Ownership  Cost 

WL  =  Whole  Life  Engineering  (Specialty  Engineering) 


Acronyms  -  Raytheon 

CMMI  DEV  and  SVC  Process  Areas 


CAM  =  Capacity  and  Availability  Management 
CAR  =  Causal  Analysis  and  Resolution 
CM  =  Configuration  Management 
DAR  =  Decision  Analysis  and  Resolution 
IPM  =  Integrated  Project  Management 
IRP  =  Incident  Resolution  and  Prevention 
IWM  =  Integrated  Work  Management 
MA  =  Measurement  and  Analysis 
OPD  =  Organizational  Process  Definition 
OPF  =  Organizational  Process  Focus 

OPM  =  Organizational  Performance 
Management 

OPP  =  Organizational  Process  Performance 

OT  =  Organizational  Training 

PI  =  Product  Integration 

PMC  =  Project  Monitor  and  Control 

PP  =  Project  Planning 

PPQA  =  Process  and  Product  Quality 
Assurance 


QPM  =  Quantitative  Project/Work  Management 

QWM  =  Quantitative  Work  Management 

RD  =  Requirements  Development 

REQM  =  Requirements  Management 

RISKM  =  Risk  Management 

SAM  =  Supplier  Agreement  Management 

SCON  =  Service  Continuity 

SD  =  Service  Delivery 

SSD  =  Service  System  Development 

SST  =  Service  System  Transition 

STSM  =  Strategic  Service  Management 

TS  =  Technical  Solutions 

VAL  =  Validation 

VER  =  Verification 

WMC  =  Work  Monitor  and  Control 

WP  =  Work  Planning 
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Writing  Requirements 

Properly 


2011  NDIACMMI  Conference  Presented  by 

Denver,  Colorado  Al  Florence 

The  MITRE  Corporation 


The  authors’  affiliation  with  The  MITRE  Corporation  is  provided  for  identification  purposes  only,  and  is  not  intended  to 
convey  or  imply  MITRE's  concurrence  with,  or  support  for,  the  positions,  opinions  or  view  points  expressed  by  this  presenter. 


Class  Participation 


Determine  the  problems  with  these  2  requirements: 


•  All  computer-resident  information  that  is  sensitive  shall  have  system 
access  controls.  Access  controls  shall  be  consistent  with  the 
information  being  protected  and  the  computer  system  hosting  the 
data. 


The  interval  for  propagating  changes  to  suppliers  shall  be  configurable. 


2 


Introduction  iofs 


A  Government  agency,  while  re-developing  legacy  systems, 
reversed  engineered  the  existing  requirements. 

The  examples  represent  several  legacy  systems  that  are  in  the 
process  of  redevelopment  in  a  modernization  effort. 

The  examples  depict  only  the  requirements  effort  -  they  do  not 
reflect  any  other  lifecycle  activities:  design,  implementation,  test 
operation. 


Introduction  2 of 3 


•  It  needs  to  be  noted  that  requirements  do  not  “live  alone” 

-  They  depend  on  other  requirements  and/or 

-  on  clarifying  comments 

to  present  a  complete  view  of  the  functionality  associated  with  a 
related  set  of  requirements. 

•  A  related  set  of  functional  requirements  may  be  introduced  with  a 
preamble  describing  the  capability  of  the  functional  set. 

-  The  preamble  does  not  itself  establish  requirements;  this  is  done 
later  in  the  requirements’  specifications. 

•  Some  requirements  may  be  amplified  with  clarifying  comments  which 
are,  again,  not  part  of  the  requirements,  but  add  understandability. 
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Introduction  3 of 3 


Some  requirements  are  documented  sequentially  with  the 
requirements  stated  first  setting  the  “stage”  for  the  following 
requirements  which  add  more  and  more  capability. 

-  The  later  stated  requirements  depend  on  the  earlier 
requirements  to  complete  their  functionally. 

-  An  example  may  be  the  use  of  the  word  “processing”.  If  the 
processing  of  a  functional  set  of  related  requirements  has 
been  described  in  earlier  requirements  the  later  requirements 
may  amplify  and/or  reference  the  processing  without  having  to 

restate  the  processing. 


Critical  Attributes  iofs 


The  following  are  some  critical  attributes  that  requirements 
must  adhere  to: 

--3^  Completeness:  Requirements  should  be  as  complete  as  possible. 


(They  should  reflect  system  objectives  and  specify  the  relationship  between  the 
software  and  the  rest  of  the  subsystems.) 


Traceability:  Each  requirement  must  be  traceable  to  some 
underlying  source,  such  as  a  system-level  requirement. 

(Each  requirement  should  have  a  unique  identifier  so  that  the  software  design, 
code,  and  test  plans  can  be  precisely  traced  back  to  the  requirement.) 


Testability:  All  requirements  must  be  testable  in  order  to 

demonstrate  that  the  software  end  product  satisfies  its 
requirements. 

(In  order  for  requirements  to  be  testable  they  must  be  specific,  unambiguous, 
and  quantitative  whenever  possible.  Avoid  vague,  general  statements.) 
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Critical  Attributes  2  of  3 


Consistency:  Requirements  must  be  consistent  with  each  other;  no 
requirement  should  conflict  with  any  other  requirement. 

^"3  (Requirements  should  be  checked  by  examining  all  requirements  in  relation  to  each 
other  for  consistency  and  compatibility.) 


Feasibility:  Each  requirement  must  represent  a  feasible  representation. 

(Requirements  that  have  questionable  feasibility  should  be  analyzed  during 
requirements  analysis  to  prove  their  feasibility.) 


Unique  identification:  Uniquely  identifying  each  requirement  is 
essential  if  requirements  are  to  be  traceable  and  testable. 

(Uniqueness  also  helps  in  stating  requirements  in  a  clear  and  consistent  fashion.) 
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Critical  Attributes  3  of  3 


— - — 


Design  Free:  Software  requirements  should  be  specified  at  a 
requirements  level  not  at  a  design  level. 

(The  approach  should  be  to  describe  the  software  requirement  functionally  from  a 
system  point  of  view,  not  from  a  software  design  point-of-view,  i.e.  describe  the 
system  functions  that  the  software  must  satisfy.  A  requirement  reflects  “what”  the 
software  shall  accomplish  while  the  design  reflects  “how”  the  requirement  is 
implemented.) 


Use  of  “shall”  and  related  words:  In  specifications,  the  use  of  the 
word  "shall"  indicates  a  binding  provision. 

(Binding  provisions  must  be  implemented  by  users  of  specifications.  To  state  non¬ 
binding  provisions,  use  "should"  or  "may".  Use  "will"  to  express  a  declaration  of 
purpose  (e.g.,  "The  Government  will  furnish..."),  or  to  express  future  tense.2) 
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Ambiguity 

Requirements  must  be  written  in  a  clear,  concise  and  unambiguous 
fashion 


Words  and  phrases  that  may  have  confusing  and  multiple 
interpretations  must  be  avoided 


-  Adequate 

-  Ad  hoc 

-  All 

-  Always 

-  Appropriate 

-  Clearly 

-  Easy 

-  Existing 

-  Fast 

-  Flexible 

-  Future 

-  If  required 

-  Immediately 

-  Large 

-  Light 


-  Limited 

-  Near  real  time 

-  Periodic 

-  Portable 

-  Rapid  A|so: 

-  Several  http://www.ppi-int.eom/newsletter/SyEN-017.php#artide 

-  Slow 

-  Small 

-  Sometimes 

-  State  of  the  art 

-  Sufficient 

-  Usable 

-  User-friendly 

-  Weight 

-  When  required 
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Examples 


•  With  domain  knowledge  of  the  system,  several  teams  reverse- 

engineered  and  defined  requirements. 

•  They  represented: 

-  the  users 

-  the  contractors 

-  the  acquisition  organization 

•  This  author  was  assigned  as  a  consultant  to  guide  the  teams  in  the 
proper  specification  of  requirements. 

•  The  following  examples  show  some  of  the  requirements: 

-  as  initially  specified  by  the  teams 

-  followed  by  this  author’s  critique  (against  the  critical  attributes) 

-  and  as  re-specified  based  on  the  critique 
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Example  1 


Initial  specification: 

Software  will  not  be  loaded  from  unknown  sources  onto  the  system  without 
first  having  the  software  tested  and  approved. 

Critique: 

•  If  it’s  tested  and  approved,  can  it  be  loaded  from  unknown  sources? 

•  If  the  source  is  known,  can  it  be  loaded  without  being  tested  and  approved? 

•  Requirement  is  ambiguous  and  stated  as  a  negative  requirement,  which 
makes  it  difficult  to  implement  and  test. 

•  A  unique  identifier  is  not  provided,  which  makes  it  difficult  to  trace. 

•  The  word  “shall”  is  missing. 

Re-specification: 

3. 2. 5. 2  Software  shall  be  loaded  onto  the  operational  system  only  after  it  has 
been  tested  and  approved. 
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Example  2 


Initial  specification: 

3.4. 6. 3  The  system  shall  prevent  processing  of  duplicate  electronic  files  by 
checking  a  new  SDATE  record.  An  e-mail  message  shall  be  sent. 

Critique:  . 

•  Two  “shalls”  under  one  requirement  number. 

•  Vague  requirement:  need  to  define  the  e-mail  message. 

•  The  requirement  has  design  implications,  SDATE  record. 

•  A  requirement  should  specify  what  the  data  in  the  record  are  and  not 
the  name  of  the  record  as  it  exists  in  the  design  and  implementation.. 

•  As  specified  it  cannot  be  implemented  or  tested. 

Re-specification: 

3. 4. 6. 3  The  system  shall: 

a.  prevent  processing  of  duplicate  electronic  files  by  checking  the 
date  and  time  of  the  submission,  and 

b.  send  the  following  e-mail  message: 

1.  request  updated  submission  date  and  time,  if  necessary,  and 

2.  the  processing  was  successful,  when  successful. 
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Example  3  10F2 


Initial  specification: 

3. 2. 5. 7  The  system  shall  process  two  new  fields  (provides  production  count 
balancing  info  to  states)  at  the  end-of-state  record. 

Critique: 

•  This  requirement  cannot  be  implemented  or  tested. 

•  It  is  incomplete.  What  are  the  two  new  fields? 

•  “Info”  should  be  spelled  out. 

Re-specification: 

3. 2. 5. 7  The  system  shall  provide  the  following  data  items  (provides  production 
count  balancing  information  to  states)  at  the  end-of-state  record: 

a.  SDATE,  and 

b.  YR-TO-DATE-COUNT 
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Example  3  2  of 2 


Re-Critique: 

•  This  rewrite  has  design  implications  SDATE  record  and  YR-TO-DATE- 
COUNT. 

•  From  a  requirements  viewpoint  it  should  specify  what  the  data  in  the 
records  are,  not  the  name  of  the  record  as  it  exists  in  the  design  and 
implementation. 

Re-Re-Specification; 

3. 2. 5. 7  The  system  shall  provide  the  following  data  items  (provides  production 
count  balancing  information  to  states)  at  the  end-of-state  record: 

a.  submission  date  and  time,  and 

b.  year-to-date  totals. 


14 


Example  4 


Initial  specification: 

3. 2. 5. 9  All  computer-resident  information  that  is  sensitive  shall  have  system 

access  controls.  Access  controls  shall  be  consistent  with  the  information 
being  protected  and  the  computer  system  hosting  the  data. 

Critique: 

•  Two  “shalls”  under  one  identifier. 

•  The  requirement  is  vague  and  incomplete.  Need  to  identify  the  sensitive 
information. 

•  What  does  “consistent”  mean? 

•  As  specified  it  cannot  be  implemented  or  tested. 

Re-specification: 

3. 2. 5. 9  All  sensitive  computer-resident  information  shall  have  system  access 
controls,  consistent  with  the  level  of  protection.  (Reference  Sensitive 
Information,  Table  5.4.1  and  Level  of  Protection  for  Sensitive  Information, 
Table  5.4.2) 
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Example  5 


Initial  specification: 

3.3.2. 1  The  system  shall  have  no  single  point  failures. 

Critique: 

•  This  is  an  ambiguous  requirement.  Needs  identification  of  what 
components  and/or  functions  the  “no  single  point  failures”  applies  to. 

•  As  specified  it  cannot  be  implemented  or  tested. 

Re-specification: 

3.3.2. 1  The  following  system  components  shall  have  no  single  point  failures 

a.  host  servers, 

b.  networks, 

c.  network  routers, 

d.  access  servers, 

e.  hubs, 

f.  switches, 

g.  firewalls,  and 

h.  storage  devices. 


Example  6 


Initial  specification: 

3.2.7. 1  The  system  shall  purge  state  control  records  and  files  that  are  older  than 
the  operator  or  technical  user-specified  retention  period. 

Critique: 

•  Requirement  is  incomplete  and  vague  without  specifying  the  retention 
period  or  providing  a  reference  as  to  where  the  information  can  be  obtained. 

•  Requirement  cannot  be  implemented  or  tested  as  stated. 

Re-specification: 

3.2.7. 1  The  system  shall  purge  state  control  records  and  files  that  are  older  than 
the  retention  period  input  into  the  system  by  either  the: 

a.  operator,  or 

b.  technical  user. 
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Example  7  2 


Initial  specification: 

3. 2. 6. 3  The  system  shall  receive  and  process  state  return  data  from  the  State 
Processing  Subsystem.  The  system  shall  provide  maintenance  of  the 
state  data  files  and  generate  various  reports. 

Critique: 

•  Two  “shalls”  under  one  requirement  number  and  multiple  requirements  in 
the  specification. 

•  The  word  “process”  in  the  first  shall  is  vague.  Need  to  define  the 
processing  required. 

•  The  second  “shall”  does  not  provide  for  valid  requirements;  they  cannot  be 
implemented  or  tested  as  stated. 

-  Needs  identification  of  type/amount  of  maintenance  required. 

-  “various  reports”  is  ambiguous. 
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Example  7  2  of  2 


Re-specification: 

3. 2. 6. 3  The  system  shall  receive: 

a.  production  data  that  contains  data  from  multiple  states,  and 

b.  state  total  amount  for  one  or  more  states, 
extracted  by  the  Returns  Processing  Subsystem. 

3. 2. 6.4  The  system  shall  parse  multi-state  data  to  respective  state  files. 

3. 2. 6. 5  The  system  shall  display  a  summary  screen  reporting  the  results  of 
processing  for  each  state  containing: 

a.  state  totals, 

b.  state  generic  totals,  and 

c.  state  unformatted  totals. 
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Example  8 


Initial  specification: 

3.2.7. 1  The  system  shall  not  prevent  the  individuals  from  entering  the  year  for 
which  they  intend  the  payment,  but  shall  provide  a  check-point  for  them 
to  ensure  that  they  are  not  making  a  mistake  in  entering  the  correct 
year. 

Critique: 

•  This  is  a  negative  requirement,  negative  requirements  should  not  be 
specified.  They  cannot  be  implemented. 

•  A  requirement  should  have  all  conditions  that  are  required.  If  conditions  are 
not  required  they  will  not  be  implemented. 

•  Two  “shalls”  under  one  requirement  number. 

•  Suggest  that  this  requirement  be  structured  in  a  positive  fashion. 

Re-specification: 

3.2.7. 1  The  system  shall: 

a.  allow  individuals  to  enter  the  payment  year,  and 

b.  provide  a  check-point  to  ensure  that  individuals  enter  the  correct 

payment  year.  ?ft 


Example  9  10F2 


Initial  specification: 

After  the  system  receives  the  Validation  file,  the  system  shall: 

•  notify  the  individual  about  acceptance  or  rejection. 

•  the  acceptance  file  must  contain  the  name  and  ZIP  code  of  the 
individual. 

•  rejected  validation  request  must  include  the  Reason  Code. 

Critique: 

•  The  second  and  third  bullets  don’t  make  sense,  try  to  read  them  as 
such: 

-  the  system  shall  the  acceptance  file  must... 

-  the  system  shall  rejected  Validation... 

•  Use  of  both  “shall”  and  “must”. 

•  No  unique  identifier,  use  of  bullets.  Bullets  cannot  be  traced. 

•  This  requirement  is  ambiguous  and  cannot  be  implemented  or  tested. 
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Example  9  2  of 2 


Re-specification: 

3. 2. 7. 3  When  the  system  receives  a  validation  file,  the  system  shall: 

a.  reject  the  file  if  it  does  not  contain  the  individuals: 

1.  name,  or 

2.  ZIP  code,  and 

b.  notify  the  individual  about  acceptance  or  rejection  with  a 
reason  code.  (Reference  Reason  Code,  Table  5.4.8) 
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Example  10 


Initial  specification: 

3. 2. 8. 2  The  enrollment  process  shall  take  from  one  to  ten  calendar  days  to 
complete  for  all  payment  types. 

3. 2. 8. 3  The  enrollment  process  shall  take  no  more  than  three  days  to 
complete  for: 

a.  credit  payment,  and/or 

b.  note  payment. 

Critique: 

These  requirements  are  inconsistent  and  in  conflict  with  each  other. 

Re-specification: 

3. 2. 8. 2  The  enrollment  process  shall  take: 

a.  one  to  three  calendar  days  to  complete  for: 

1.  credit  payment,  and 

2.  note  payment,  and 

b.  one  to  ten  calendar  days  to  complete  for  all  other  payment 
types. 
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Example  11 


Initial  specification: 

3.2.9. 1  When  doing  calculations  the  software  shall  produce  correct  results. 

Critique: 

•  Really?  This  is  not  a  requirement. 

•  This  type  of  requirements  should  not  be  specified! 

•  It  should  be  deleted. 

Re-specification: 

Requirement  deleted. 
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Summary 


The  teams  identified  over  1000  requirements. 

The  issues  with  their  initial  specification  represented  the  entire 
spectrum  of  the  following  critical  attributes: 

-  feasibility 

-  completeness 

-  traceability 

-  testability 

-  consistency 


unique 

identification 


design  free 
use  of  shalls 


The  teams  were  receptive  to  the  critique,  resolved  issues  and 
implemented  the  recommendations  willingly. 


•  The  requirements  resulting  from  this  effort  were: 

-  reviewed  with  senior  management 

-  accepted  as  specified 

-  baselined,  and 

-  allocated  to  development  teams  for  implementation. 


Conclusion 


•  If  sufficient  time  and  proper  effort  is  taken  to  validate  requirements 
against  critical  attributes  during  their  definition  and  specification, 
software  projects  will  improve  their  probability  of  success  considerably. 

•  If  this  is  not  done,  projects  pay  the  consequences  during 
implementation,  integration  and  test  -  not  to  mention  during  operation. 


But  you  knew  that,  didn’t  you? 

(I  hope!) 


I 
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Contact  Information 


Al  Florence 

florence@mitre.  org 
703  395  8700  -  Cell 
303  955  2286  -  Home 
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CMMI®  Without  Appraisals 


Alan  Gellis 

alan.h.gellis@boeing.com 

Defense,  Space  &  Security 
Boeing  Military  Aircraft 
Mobility  -  Philadelphia 
November  17,  2011 


BOEING  is  a  trademark  of  Boeing  Management  Company. 
Copyright© 201 1  Boeing.  All  rights  reserved. 


CMMI®  Without  Appraisals 


■  At  this  time  of  economic  challenges,  many  companies  are  looking 
at  the  cost  of  CMMI  appraisals*  as  an  area  that  needs  to  be 
evaluated.  This  paper  explores  some  of  the  issues,  challenges, 
and  decisions  involved  in  that  evaluation  and  the  resulting  course 
of  action. 

■  Because  there  are  so  many  variations  in  company  situations 
regarding  market  requirements,  competition,  their  own 
capabilities,  and  their  objectives,  there  is  no  single  approach  that 
is  best  for  all. 

■  As  a  result,  this  paper  is  intended  to  explore  the  issues  and 
choices  in  a  way  that  may  make  it  easier  to  reach  sound  decisions 
in  individual  situations. 


*As  used  in  this  paper,  “Appraisals”  refers  to  formal,  Class  A 
SCAMPI  Appraisals. 


BOEING  is  a  trademark  of  Boeing  Management  Company 
Copyright  ©2011  Boeing.  All  rights  reserved. 
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Starting  Point  -  If  you  have  a  rating 


■  Some  companies  currently  have  a  rating  achieved 
through  a  formal  SCAMPI  Class  A  appraisal.  The 
issues  they  face  include: 

■  Maintaining  the  appraised  level 

■  Seeking  a  higher  or  lower  level 

■  Terminating  appraisal  activities 

■  Expanding  the  area  covered  by  their  appraisal 

■  Reducing  the  area  covered  by  their  appraisal 

■  Maintaining  the  capability  to  pass  an  appraisal  in  the 
future  if  the  need  arises,  while  not  having  one  now 


BOEING  is  a  trademark  of  Boeing  Management  Company 
Copyright  ©2011  Boeing.  All  rights  reserved. 


3  of  20 


Starting  Point  -  Without  a  rating 


■  Some  companies  have  not  achieved  a  rating  through  a 
formal  appraisal,  or  no  longer  have  one.  The  issues 
they  face  include: 

■  Evaluating  their  processes  to  see  if  they  would  qualify  for 
a  rating 

■  Evaluating  their  performance  to  see  if  pursuit  of  a  rating 
would  be  cost-effective  as  a  way  to  improve 

■  Evaluating  their  market,  customer  requirements,  and 
competition  to  see  if  a  rating  is  necessary  now  or  in  the 
future 

■  Evaluating  other  methods  and  standards  as  a  way  to 
improve  performance 
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Evaluating  the  Market  -  First  Steps 


Before  the  evaluation  of  the  market  becomes  a  prime  factor,  a 
company  must  go  through  a  thought  process  that  evaluates 
these  factors: 

■  Process  improvements  are  necessary 

■  CMMI  is  a  good  way  to  achieve  improvements 

■  CMMI  requires  an  evaluation  to  be  effective 

■  A  Class  A  appraisal  is  a  better  evaluation  than  a  more  informal 
appraisal,  or  an  unofficial  review 

■  Although  process  improvements  and  CMMI  may  eventually 
result  in  lower  cost,  there  is  an  up-front  cost  to  pursue  an 
appraisal  and  formal  rating.  This  cost  will  depend  on  how 
much  will  need  to  be  changed  to  achieve  the  rating. 

■  There  may  be  better  uses  for  the  investment  -  e.g.,  factory 
improvements 
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Evaluating  the  Market 


■  Once  a  company  decides  that  market  factors  are  “drivers”  for  a 
decision  to  pursue  an  appraised  rating,  it  must  consider  these 
factors: 

■  Does  a  customer  currently  require  a  rating? 

■  Does  a  customer  require  use  of  CMMI  equivalent  processes 
(many  phrases  for  this)  without  appraisals? 

-  Not  just  mapping 

-  Will  a  rating  be  a  discriminator? 

■  Do  competitors  have  appraised  ratings? 

■  Are  current  processes  “good  enough”  to  get  a  rating? 

■  What  are  cost  and  time  to  get  to  a  rating,  if  needed? 

■  What  is  risk  of  having  to  forego  business  if  no  rating?  [Use  risk 
management  process] 
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If  the  Market  is  not  a  factor 


■  Is  the  business  software  intensive? 

■  If  Yes,  CMMI  more  likely  to  be  necessary 

■  If  No,  CMMI  less  likely  to  be  necessary,  but  may  still  add 
value 

■  Is  cost /  quality  improvement  a  competitive  or  market 
factor? 

■  If  Yes,  CMMI  may  be  a  good  choice  -  must  evaluate 

■  If  you  say  No,  you  probably  don’t  understand  your 
situation 

■  Can  advantages  of  CMMI  be  achieved  without 
appraisals?  Whole  set  of  issues! 
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Can  advantages  of  CMMI  be  achieved  without 

appraisals? 


■  If  a  company  has  superlative  processes  and  execution,  CMMI 
won’t  add  -  but  may  make  management  easier 

■  If  a  company  has  less  competent  processes  and  execution,  CMMI 
can  provide  advantages 

■  Better  Quality;  lower  cost;  customer  satisfaction 

■  Cost /  benefit  test 

■  If  a  company’s  processes  and  execution  are  inadequate,  anecdotal 
evidence  suggests  that  following  just  a  few  CMMI  practices  may 
help  -  question  of  how  much 

■  Following  “Good  Requirements  Definition”  and  Peer  Reviews 
may  be  an  “80%  solution”  at  less  than  80%  cost  -  numbers  are 
hard  to  substantiate,  and  would  depend  on  starting  point 

■  Other  initiatives  may  be  alternatives 

■  ISO/AS9100,  6  Sigma,  Baldridge 

■  May  cost  as  much;  may  have  less  “market  value” 
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Evaluating  the  “Real”  cost  of  appraisals 


■  People  tend  to  lump  related  work  into  cost  of  appraisals; 

1.  Conducting  the  appraisal,  itself,  including  cost  of 
appraiser 

-  There  is  a  real  cost  of  an  appraisal,  but  it  can  be  managed  or 
reduced 

2.  Developing  compliant  processes 

■  Should  be  done,  even  without  appraisals 

3.  Managing  compliant  processes 

■  Should  be  done,  even  without  appraisals 

■  CMMI  does  not  require  micromanagement,  excessive  detail,  or 
“churning”  on  improvements  and  details,  but  many  people 
mistakenly  think  it  does 

4.  Process  Training 

■  If  done  right,  no  extra  costs  -  skill  training  is  separate  issue 

5.  Preparation  for  appraisals,  including  artifact  “systems” 

■  Many  opportunities  to  reduce  costs,  use  existing  systems 
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Going  for  “a  number”  vs.  real  improvement 


■  Despite  all  the  advice  against  it,  many  companies  go 
for  an  appraised  level  as  a  “qualification”  they  can 
publicize,  and  do  not  work  to  get  real  improvement 

■  Internal  incentives  often  drive  this  behavior 

■  These  companies  don’t  get  the  real  advantages  of 
CMMI  even  with  an  appraisal,  so  they  will  get  even 
fewer  without  one 

■  To  be  useful,  we  need  to  confine  our  study  to 
companies  seeking  real  cost  and  quality  improvement 
through  process  improvements,  and  their  evaluation  of 
the  pros  and  cons  of  having  formal  appraisals 
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“Lean”  vs.  CMMI 


■  It  is  possible  to  pursue  CMMI,  get  an  appraised  Level  5,  have 
effective  and  well  managed  processes,  and  still  have  high  costs 

■  Lean  is  an  additional  discipline  or  technique  that  can  help  reduce 
costs  and  provide  greater  efficiency.  Reduced  costs  make  the 
cost/benefits  test  for  appraisals  easier  to  pass 

■  The  best  and  most  cost  effective  results  occur  when  both  CMMI 
and  Lean  are  used  together 

■  This  is  not  easy  to  do  and  it  can  take  a  long  time  to  get  optimum 
results 

■  Dependencies: 

■  Level  and  rate  of  expenditure  on  these  initiatives 

■  Leaders’  creativity  and  authority  to  make  changes 

■  Cannot  properly  evaluate  choices  regarding  appraisals  without 
considering  Lean  component  in  both  processes  and  appraisals 
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Large/Diverse  company  considerations 


■  A  large/diverse  company  may  have  very  different  operations  at 
different  locations: 

■  Some  may  do  development  where  CMMI  is  appropriate 

■  Some  may  only  do  “cookie  cutter”  manufacturing  where 
traditional  quality  control  approaches  work  better 

■  A  large  company  has  opportunity  to  develop  common  processes 
at  low  cost  on  a  small  scale,  and  then  deploy  them  across  the 
company  -  but  one  size  does  not  always  fit  all 

■  A  company  may  have  some  areas  with  customers  and  markets  that 
require  ratings,  and  others  that  do  not 

■  If  not  careful,  a  large  company  can  find  increased  bureaucracy  and 
overhead  costs  across  its  businesses,  rather  than  efficiencies  of 
scale 

■  A  strategy  is  needed  that  matches  the  company’s  needs  -  no 
universal  approach  here 
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Different  CMMI  Constellations 


■  Most  of  the  material  in  this  presentation  was  written  with  CMMI  for 
Development  in  mind,  but  it  would  also  apply,  in  general,  to  CMMI 
for  Acquisition  and  CMMI  for  Services,  with  a  little  change  in 
emphasis 

■  The  main  issue  is  whether  the  activity  is  being  pursued  and 
performed  via  disciplined  processes,  and  whether  it  can  benefit 
from  an  organized  approach  to  that  effort 

■  Historically,  it  has  taken  many  companies  several  years  to  get  their 
process  sets  developed  and  performed  with  sufficient  skill  and 
discipline  to  be  ready  for  a  CMMI  appraisal.  This  would  be  the 
case  with  each  of  the  Constellations,  depending  on  individual 
company  situations 

■  The  first  step  is  a  determination  that  a  company  wants  to  work  to 
processes  at  the  level  of  sophistication  of  the  CMMI 
Constellations,  and  then  embark  on  a  journey  of  improvement 
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Is  a  simple  checklist  possible? 


■  It  would  be  nice  to  have  a  simple  checklist  to  see  whether  a 
company  should  seek  an  appraised  CMMI  level  through  a  formal 
appraisal,  but  this  is  not  possible 

■  Except  for  the  situation  where  a  customer  requires  it,  all  of  the 
other  situations  involve  some  evaluation  and  discretion,  and  a 
checklist  is  inadequate  for  them 

■  If  a  company  is  not  interested  in  process  discipline  as  a  way  to 
manage  and  improve  its  business,  CMMI  never  comes  up,  so  we 
can  ignore  those  cases 

■  If  a  company  only  makes  hardware  and  is  convinced  that  a 
standard  such  as  ISO  9001  or  AS9100  is  adequate,  CMMI  also  does 
not  come  up 

■  In  all  other  cases,  we  are  dealing  with  cost /  benefit/  risk  issues, 
and  must  do  an  evaluation  accordingly 
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The  risk  situation 


■  If  there  is  a  significant  risk  that  a  customer  will  require  a  CMMI 
level  as  a  condition  to  bid,  and  that  business  is  important  to  a 
company,  it  is  hard  to  justify  not  pursuing  the  necessary  appraised 
level.  The  issue  becomes  how  to  pursue  it  efficiently,  and  that 
gets  into  Lean  considerations. 

■  If  a  customer  requires  “CMMI-like”  processes,  but  no  appraisal, 
and  competitors  have  ratings,  there  is  a  risk  of  self-deception  by 
thinking  your  processes  are  “good  enough”  when  they  are  not.  In 
this  case,  not  having  an  appraised  level  presents  a  higher  risk  than 
spending  the  money  on  an  appraisal  and  not  needing  it.  This  is  a 
risk  evaluation/  management  situation. 

■  If  a  customer  requires  a  CMMI  rating  within  some  period  after 
award,  the  question  becomes,  “Why  wait?”  This  is  not  just  a  cost 
issue,  but  recognition  that  the  transition  to  CMMI  processes  and 
the  appraisal  that  follows  can  add  a  measure  of  disruption  that  can 
adversely  affect  contract  performance.  However,  this  is  a 
judgment  call.  This  is  also  a  risk  evaluation/  management 
situation. 
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The  hard  question 


■  It  is  easy  for  companies  not  faced  with  any  customer 
CMMI  requirements  to  say  something  such  as  “We 
have  good  processes  that  are  CMMI  ‘compliant’,  so  if 
we  just  follow  them  without  having  an  appraisal,  we  get 
all  of  the  advantages  and  we  don’t  have  to  incur  the 
costs  of  an  appraisal.” 

■  But  - 

■  The  processes  are  usually  not  compliant  and  not 
managed  as  well  as  they  should  be,  so  this  is  a  form  of 
self-deception.  In  practice,  a  CMMI  appraisal  may  be 
the  only  good  way  to  validate  your  processes 

■  That  gets  to  the  hard  question,  “Can  we  afford  not  to 
have  a  CMMI  appraisal?” 
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The  complex  answer  to  the  hard  question 


■  If  a  company  has  asked  itself,  “Can  we  afford  not  to 
have  a  CMMI  appraisal?”,  then  the  answer  depends  on: 

■  Having  good  processes  and  process  management 

■  Having  a  corporate  will  to  do  better,  and  to  do  it  in  a 
disciplined  way 

■  Having  the  desire  and  ability  to  apply  a  Lean  approach  so 
that  an  appraisal  will  cost  no  more  than  is  absolutely 
necessary 

■  If  all  of  these  are  present,  it  probably  pays  to  have  the 
appraisal 

■  If  not  all  are  present,  it  probably  will  be  costly  to  have 
an  appraisal,  and  the  choice  of  having  the  appraisal  will 
depend  on  whether  the  company  wants  to  choose  this 
path  toward  process  improvement 
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Conclusion 


■  Except  for  the  situation  where  having  an  appraised  CMMI  rating  is  a 
customer’s  condition  for  getting  necessary  new  business,  there  are 
choices  involved  in  deciding  whether  to  undertake  the  appraisal 

■  Those  choices  depend  on  a  number  of  factors,  including 

■  The  nature  of  the  business  (e.g.,  software  or  not) 

■  The  current  state  of  their  processes  (good  or  not) 

■  The  corporate  will  to  improve,  and  to  pursue  CMMI 

■  The  ability  to  apply  a  Lean  approach  to  processes  and 
appraisals  to  reduce  the  costs  of  appraisals 

■  Whether  the  company  will  be  content  with  self-appraisals 
of  their  processes,  with  the  recognition  that  they  almost 
certainly  won’t  be  as  good  as  if  validated  by  a  formal 
appraisal 
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Conclusion  (continued) 


■  The  Thought  process: 

"  You  need  to  improve  both  processes  and  execution 

■  That  can  be  done  either  with  or  without  help  from  a  proven  system 

-  Without  the  help  -  inconsistent  results,  hard  to  maintain 

-  With  help  of  a  proven  system  -  better  results  if  done  right 

-  CMMI  can  be  the  system  of  choice,  but  must  be  done  right.  Type  of  business  can 
influence  this  choice. 

■  CMMI  can  be  used  with  or  without  appraisals 

-  Hard  to  validate  approach  without  appraisals  -  temptation  is  to  declare  victory  with 
insufficient  data.  Management  must  make  up  for  what  is  lost  with  no  appraisals.  Is  there 
really  a  saving  this  way? 

-  Having  appraisals  in  one  area  and  using  same  processes  elsewhere  ignores  the  factor  of 
execution  that  is  improved  with  appraisals 

-  Appraisals  give  basis  for  correcting  where  needed;  validation 

-  Real  cost  of  appraisals  must  consider  both  effort  and  value  added 

-  Cost  can  be  reduced  with  a  Lean  approach.  (This  is  not  necessarily  easy  or  quick,  but 
should  be  considered  in  the  thought  process.) 

-  Using  CMMI  by  adopting  only  a  few  practices  or  processes  will  add  a  little  value,  but  will 
still  leave  considerable  room  for  improvement 

-  Why  do  this,  when  full  adoption  gives  better  results? 

■  Path  to  improvement  must  be  selected 
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Conclusion  (continued) 


■  There  is  no  single  answer  that  fits  all  situations 

■  Arriving  at  the  right  answer  depends  on  going  through 
an  evaluation  process 

■  It  is  hoped  that  this  presentation  will  assist  those 
considering  whether  to  have  a  formal  appraisal  to 
arrive  at  the  right  decision  for  them 

■  Just  remember  that  CMMI  without  formal  appraisals  is 
not  really  CMMI,  and  if  this  is  your  choice,  you  must  be 
prepared  to  live  with  the  shortcomings 
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Systonomy  background 


systonomy 


Founded  in  April  1999,  Systonomy  is  dedicated  to  the  application  of  Empirical  and  Experimental 
Software  engineering,  Six  Sigma  and  DFSS  for  IT  and  Software  Development  from  real-time  and 
embedded  systems  to  Management  Information  Systems  (MIS)  including  the  implementation  and 
integration  of  COTS,  EAI,  ERP  systems,  CRM,  Financial  Systems  etc. 
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Systonomy  has  devised  a  unique  Six  Sigma  and  DFSS  framework  for  IT  and  Software 
Engineering  that  is  at  the  forefront  of  current  knowledge  and  is  investing  heavily  in  research  into 
new  methods.  Our  training  has  been  designed  from  the  ground  up  as  an  IT/Software  Six  Sigma 
and  DFSS  training  programme  and  is  not  a  superficial  modification  of  manufacturing  or 
transactional  Six  Sigma.  Our  adaptive  approach  offers  our  clients  an  innovative  and  low  risk 
move  from  defensive  strategies  to  those  of  growth. 


Our  Change  Managers,  Advisors,  Engineers,  Black  Belts,  Master  Black  Belts  and 
Instructors  are  IT  professionals  first  and  statisticians  second 
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Software  Engineering  Culture 


What  do  these  have  in  common? 
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Fiction  Reality 
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Spontaneous  Processes  and  Entropy 


#■  The  idea  that  Entropy  =  Disorder  is 
an  obsolete  and  misleading 
concept 


#■  Entropy  is  about  probability  of  all 
possible  choices,  ...  not  whether  a 
given  arrangement  looks  neat  or 
messy. 


♦  Spontaneous  processes  go 
towards  state  with  the  most 
possible  options 

■  “Driven”  by  simple  statistics 

■  Random  process,  not  actually  driven 

0  Entropy  is  the  number  of  available 
options 

Statistical  Definition  of  Entropy 

■  S=  k  ln(W) 

■  W  =  #of  available  states  of  equal  energy 

■  (Entropy  =  Delocalization  of  energy) 


3  Shirts:  Red,  Blue,  Green 


Shirts  limited  to  pile  in 
drawer 


W  =  6 

red  /  blue  /  green, 
blue  /  red  /  green,  etc. 


Shirts  anywhere 
in  room 


W  =  Lots 

red  on  chair  /  blue  on  floor, 
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Internal  and  External  Entropies 
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Maturity-based  models 
value  more  the  control  of 
Internal  Entropy 


Two  types  of  entropy: 

■  Internal  entropy 

■  External  entropy 


High  Agility  Low 

{'  Low 

!  ^ 

i  co 

i— H 

!  C 

!  r+ 

*< 

0 

< 

:  cd_ 

!  05 

I  High 

- > 

Low  External  Entropy  High 


#  EntropySolid  <  EntropyLiquid  <  EntropyGas 
<§>  Increased  Rigidity  ->  Decreased  Entropy 

#  Increased  #  Atoms  (elements)  ->  Increased  Entropy 

#  Increased  Mass  (Complexity)  ->  Increased  Entropy 


Agile  methods  value 
more  the  understanding 
of  External  Entropy 


Working  software  over  comprehensive  documentation 
Customer  collaboration  over  contract  negotiation 
Responding  to  change  over  following  a  plan 
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Maturity  versus  Capability  demystified 

The  Cow  Maturity  ModelSM 
The  Milking  Process 


X’s: 


Quality  of  the  Cows 
Grass,  Food 
Massaging  cows 
Discipline 
Sophistication 


Y’s: 


Litre/Cow/Day 
Density  of  cream/litre 
Defective  milk 


capability 


systonomy 


Maturity  and  agility  versus  Capability 


maturity  agility 

#  Maturity-based  models,  agile  methods  and  other  methods 
claim  some  kind  of  positive  relationship  between  their 
intrinsic  property  and  Capability  (performance) 

■  Maturity  and  Capability 

■  Agility  and  Capability 

■  etc 
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Superstition  is  merely  the  confusion  of  correlation 
and  causality 


systonomy 


#  Superstitious  behaviour:  believes  that  the 
action  has  an  immutable  cause-and-effect  to 
the  outcome,  whereas  the  action  might  or 
might  not  be  functional 

♦  "I  behave  this  way,  and  I  achieve  results. 
Therefore,  I  must  achieve  results  because  I 
behave  this  way.“ 

#  Superstition:  information  accepted  on  faith, 
without  personal  knowledge  or  examination. 

♦  People  pass  along  "everyone  knows"  data 
without  questioning  it,  and  others  accept  the 
superstition  as  undeniably  true 

#>  Confidence  isn't  knowledge; ....  confidence  can 
prevent  knowledge  and  innovation  from 
happening,  Unquestioned  belief  means  you 
never  measure,  never  test,  never  look  at 
alternatives 

Youfffifi 

http://www.voutube.com/watch?v=vGazvH6fQQ4 
http://www.voutube.com/watch?v=TtfQlkGwE2U&NR=1 


#  Skinner  Experiment 

■  He  deprived  pigeons  of  food  for  a  period  of  time  to 
ensure  high  responsiveness  to  anything  which  resulted 
in  food. 

■  Placed  them  in  a  cage  which  delivered  a  food  pellet 
every  15  seconds,  regardless  of  what  the  pigeon  did  or 
did  not  do. 

■  The  pigeon  cannot  do  anything  to  obtain  the  food  or 
ensure  the  supply  would  continue. 

■  The  pigeons  began  recalling  their  actions  before  the 
food  was  delivered.  Some  pigeons  turned  in  circles, 
others  tilted  their  heads,  others  tapped  their  feet, 
others  swayed  their  bodies,  and  others  tossed  their 
heads. 

■  The  pigeons  associated  these  particular  actions  with 
food  delivery  and  began  repeating  them  in  the  manner 
of  a  ritual 

#  Skinner  drew  the  following  conclusions: 

■  The  pigeon  behaves  as  if  there  were  a  causal  relation 
between  its  behaviour  and  the  supply  of  food,  although 
such  a  relation  is  lacking. 

■  A  few  accidental  connections  between  a  ritual  and 
favourable  consequences  suffice  to  set  up  and 
maintain  the  behaviour  in  spite  of  many  unreinforced 
instances 

■  Such  a  stimulus  has  reinforcing  value  and  can  set  up 
superstitious  behaviour. 

■  The  experiment  might  be  said  to  demonstrate  a  sort  of 
superstition. 

■  There  are  many  analogies  with  human  behaviour 
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Examples  of  Superstitions  in  software  engineering 
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<#■  Unfounded  opinions  and  beliefs 

■  Java  is  better  than  C  or  C++  or  C# 

■  COBOL  is  an  extinct  language  and  not  useful  for  anything 

■  Open  source  languages  are  free! 

■  CMMi  is  old  fashion,  Agile  is  great 

■  Agile  is  not  a  proper  method 

Requirements  Gathering:  ♦  Project  Planning 

The  requirements  gathering  process  has  an  Planning  is  another  inherently  speculative  activity, 

inevitable  speculative  element  to  it.  In  order  for  planning  activities  to  have  any  bearing 

on  reality  they  must  be  closely  and  interactively 
allied  to  actual  outcomes. 


Superstitions 

Common  superstition  is  that  all 
requirements  must  be  clearly  defined  before 
the  project  can  start. 

We  also  know  it  is  the  worst  time  to  define 
all  the  requirements  because  it  is  the 
furthest  point  away  from  the  time  of  use.. 


Superstition 

One  of  the  most  pervasive  superstition:  the  best 
plan  is  a  complete  plan,  the  more  detail  the  better, 
the  more  accurate  we  make  our  forecasts,  the 
more  realistic  our  plan  will  be. 

The  project  becomes  schizophrenic,  it  has  two 
independent  realities.  One  is  the  “reality”  of  the 
plan,  effort  is  expended  trying  to  force-fit  what 
really  happened  into  a  shape  that  we  pretend  it 
was  the  same  as  we  thought  would  happen. 
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SE  -  Software  Engineering  discipline 
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#■  The  term  “Software  Engineering”  was  coined  the  first  time 
in  1968  at  a  NATO  conference 

#■  The  discipline  of  SE  is,  unfortunately,  still  in  its  infancy 

#■  SE  has  reached  a  stage  that  is  more  resembling  to 
quackery  than  engineering 

#■  The  modus  operand i  of  ideas  adoption  within  SE  is  similar 
to  fashion  industry  rather  than  true  Engineering 

■  The  certainty  of  ideas  in  SE  are  judged  by  whether  people  use  the 
idea 

#■  SE  may  be  a  field  whose  progress  is  threatened  by 
analogies  and  beliefs 

■  Are  we  sure  that  our  beliefs  are  true? 

■  Which  claims  made  by  the  software  community  are  valid? 

■  Under  which  circumstances  are  they  valid? 


Sbr  (iantf  nf  ittrbirunl  ilirfiirinr 


2-4  players 
Ages: 

1 0  and  up 


TANGENT  GAMES 

IV  J  Taking  games  In  o  new  dbecHon 


Playing  time: 
30  to  60 
minutes 
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Empirical  and  Experimental 
Software  Engineering 


Empirical  Methods 


systonomy 


#  Empirical  methods  provide  us  with  insights  into  how  software 
engineering  works  in  practice  and  how  changes  to  the 
process  can  results  in  changes  to  the  outcomes  of  the 
process  (improvements) 
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Empirical  Software  Engineering 


#■  Empirical  Software  Engineering  is  a  discipline  which  attempts  to  understand 
phenomena  and  at  the  same  time  try  to  change  those  phenomena  in  order  to 
improve  them. 

#■  It  is,  therefore,  about  contemplation  and  action;  it  has  two  aims: 

■  Understand  how  software  is  actually  developed  and  maintained 

■  Understand  what  improvements  should  be  made  to  software  development  (engineering)  and 
how  these  improvements  should  be  implemented 

#■  It  promotes  empirical  evidence  as  the  primary  source  of  reliable  knowledge  to 
achieve  these  two  goals 

■  Exploratory  -  forming  hypothesis 

■  Experimental  -  involves  planned  experiment  designs  in  to  prove  causation  and  not  only 
correlation 

Empirical  =  Exploratory  +  Experimental 

#  It  is  a  necessary  technology  or  a  natural  approach  for  goal-oriented, 
sustained  process  improvement 

[RAI  07] 
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Empirical  Cycle 

Closing  the  gap  between  theory  and  practices 
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confirmation  /rejection 
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Empirical  and  Experimental  approaches  often  rely 
on  Statistical  Thinking 


systonomy 


data 


knowledge 
- > 


Statistical  Thinking 

■  All  work  is  process 

■  Variation  exist  in  any  process 

■  Understanding  and  reducing  variation  are  keys 
to  success 

Five  fundamental  habits  are  operating 
simultaneously: 

■  The  need  for  data 

■  The  importance  of  data  production 

■  The  omnipresence  of  variability  and  uncertainty 

■  The  measuring  and  modelling  of  variability  and 
uncertainty 

■  Interpreting  results  in  a  context 


©  Systonomy  Limited  -  NDIA  conference  -  CMMI  Technology  201 1  -  Denver 


page  17 


Learning. ..and  Knowledge  acquisition 
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■#  The  purpose  of  Empirical  and  statistical  Thinking  is  Learning 
and  Knowledge  acquisition 


Learning  is  not  compulsory .... 
neither  is  survival  ” 

W.  Edwards  Deming 


Empirical  and  Experimental  software  engineering  is 
for  all  professionals  who  have  (and  want  to  keep)  a 
child’s  mindset  and  ask  the  question  why? 


Software  Six  Sigma 
A  Problem  Solving  Methodology 


Is  Six  Sigma  another  fad? 
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#  YES...  If 

■  It  is  used  as  a  fagade 

■  It  is  used  as  a  label  or  a 
branding 

■  It  is  used  for  “compliance” 
purposes 

■  Statistics  are  (ab)-used  to 


justify  early  decisions 


#  NO...  If 

■  It  is  used  as  a  real  problem 
solving  methodology 

■  Improvements  solutions  are 
linked  to  the  organisations 
goals,  values  and  tangible 
benefits 

■  it  used  to  gain  insights  on  our 
practices 
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Problem  Solving  Methodology 


systonomy 


DESIGN 

OPTIMISE 

I 

r  > 

VERIFY 

^  — I 

/ - l 

i 

IMPROVE 

l 

CONTROL 

■ 

/ - > 

DEFINE 

l 

l 

■ 

t - j 

MEASURE 

■ 

l 

■ 

ANALYSE 

Six  Sigma  is  a  pragmatic  approach  to  Empirical  and  Experimental  Process 
Improvement 

KEYWORDS: 

•  Pragmatic:  Six  Sigma  is  about  solving  problems.  The  focus  is  on  business 
problems  that  cause  “pain”  and  extra  costs  to  the  organisation 

•  Experimental:  Six  Sigma  is  not  a  catalogue  of  best  practices  or  methods.  Every 
organisation  is  different  and  so  are  the  problems  they  face.  Six  Sigma  rejects  pre¬ 
defined  solutions  and  investigates  problems  to  the  level  of  their  root  causes 
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Problem  Solving  Methodology 
A  rigorous  approach  to  process  improvement 


systonomy 
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Six  Sigma  Problem  Solving  cycle 
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Statistical  Domain  Critical  x  s 
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Six  Sigma  vs.  Other  SPI  approaches 
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Six  Sigma 

Catalogue  Based  Improvement  models 

Focus  on  Problems 

Focus  on  Best  Practices  (Solutions) 

Emphasis  on  measurement  of 
process  capability 

Emphasis  on  assessment  of  process 
maturity 

Business  results  oriented 

Quality  improvement  oriented 

More  prescriptive  in  nature 

More  descriptive  in  nature 

Improvement  “is  by  experiment” 

Improvement  is  “by  the  book” 

Provides  the  “how  to”:  solve  a 
process  problem 

Provides  the  “how  to”:  manage  the 
process  according  to  best  practices 

The  two  approaches  complement  and  reinforce  each  other! 

Six  Sigma  integrates  pragmatism  into  Empirical  approaches 
without  loosing  the  scientific  rigour 
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Key  Six  Sigma  Concept:  Hidden  Factory 
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Process 

A 

B 

c 

Final  Test 

D 

f  90% 

r 

90% 

r  \ 

90% 

r  \ 

90% 

Yield 

l  Yield  J 

l  Yield  J 

Yield  J 

Rolled  Yield 

81  % 

73% 

66% 

Rolled-Throughput  Yield 


Manufacturing 
Variation  Causes  a 
"Hidden  Factory" 
Increased  Cost  - 
Lost  Capacity 

•  Wasted  Time 

•  Wasted  Money 

•  Wasted  Resources 

•  Wasted  Floor  space 


66%  *  90% 


Classical  First-Time  Yield 


DPMO  forces  you  to  look  at  the  “hidden  factory”  where  expediting,  rework  and  delays  occur, 
but  would  likely  not  show  up  in  classical  yield  metrics.  The  resulting  detail  from  DPMO 
determinations  can  then  help  to  prioritize  where  improvements  can  be  made. 
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The  hidden  Factory  in  Software  Process  development 
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Unit  of  work 

•  Function  Points 

•  use  case  points 
•.  widgets 


Delivered  unit  of  work 

•  Function  Points 

•  use  case  points 
•.  widgets 


Area  of  opportunities  (xIO6) 


Delivered  defects  per 
area  of  opportunities  (xIO6) 
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The  hidden  Factory  in  Software  Process  development 
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The  Hidden  Factory 

Defects  are  not  recorded 
prior  to  system  test 

We  are  not  recording  the 
True  Yield 

The  Box  called  Software 
Development  is  a  black 
box  and  hides  other 
defects  and  reworks  that 
could  be  avoided. 
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The  hidden  Factory  in  Software  Process  development 


systonomy 


■#  Activityj  inherited  a  widget  with  7  defects 

Activityj  introduced  3  new  defects,  but  also  detected  2  existing  defects  to  be  fixed  by 
Activity;  or  ActivityM,  ... 

♦  Verification;  detected  4  defects,  but  introduced  3  new  defects 
We  have  two  types  of  Yields: 

■  Ability  to  produce  non-defective  units  (equivalent  to  the  classical  Yield) 

■  Ability  to  detect  defects  present  at  a  given  stage  (mainly  related  to  verification  &  validation 
activities) 
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The  hidden  Factory  in  Software  Process  development 
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DIR 


DIR 


DIR 


Where  should  we 
start  from  DRE  or 
DIR? 


Hidden  Factory 


#■  The  activity  yield  and  the  total  process  yield  depends  on  two  parameters: 

■  The  Defect  Injection  Rate  (DIR) 

■  The  Defect  Removal  Effectiveness  (DRE) 
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Reality  of  Six  Sigma  and  Experimental  approaches 
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our 

#  Understanding  and  characterising  defects  provides  insight  for  process  \obse^!l 
improvement 

■  Help  prioritising  effort 

■  Provide  a  quantification  of  the  “pain”  (how  big):  Cost! 

■  Type  of  techniques  to  prevent,  contain  or  remove 

#  The  defects  data  (frequency  and  cost)  is  very  much  contextual 

#  Not  many  organisations  are  conscious  of  the  cost  of  their  defects 

#  The  theory  and  even  logic  may  dictate  that  we  should  start  from  Requirements 
and  DIR 

#  In  reality  it  is  very  difficult  to  define  what  is  a  defect  for  requirements  and  for 
Design...  Because  that  means  we  know  what  is  a  “good”  Requirement  and  a 
“good”  design.  Therefore  we  should: 

■  Start  from  defects  that  are  close  to  the  field 

■  Characterise  and  profile  defects  to  learn  about  the  process 

■  Start  from  problems  related  to  effectiveness  first  and  efficiency  second 
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Stop  the  bleeding  first... 
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Upstream  _  Downstream 

-  - ► 
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Six  Sigma 
Case  Study 


Case  Study 


systonomy 


#•  Objective  of  the  case  study  is  to  show  how: 

■  The  Six  Sigma  DMAIC  roadmap  is  a  continuous  learning  process 

■  The  problem  perception  and  formulation  keep  changing  throughout  the  entire 
DMAIC 

i  The  project  started  as:  “We  have  a  problem  with  the  testing 
process” 


<#•  Questions  to  the  audience: 

■  What  are  the  typical  problems  for  a  testing  process? 

■  Think  of  a  “pain”  and  why  would  that  be  a  pain.. 

■  How  would  you  quantify  the  problem  (“pain”)? 


Testing 

1 
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Problem  Definition  -  “Pain” 
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A  typical  dialogue... 

--  Our  Testing  process  is  not  capable 

What  do  you  mean  by  “not  capable”?  Are  there  too  many 
defects  reaching  the  customer? 

-  YES 

--  Actually  NO...  Customers  are  happy.  However  we 
spent  too  much  time  on  the  testing  process. 

Do  you  know  how  long  your  test  process  takes? 

-  NO...  35%  to  45%  of  the  development  cycle 

But  that’s  another  problem  it  is  not  capability 
(effectiveness)...  It  is  efficiency 

-  Yes,  we  want  to  reduce  the  testing  cycle 

-  May  be  we  are  testing  too  much 
--  We  were  thinking  about  automating  the  test  cases 

Yes  but  these  are  solutions... 


Cm 

^ — i — J 

W  ^  MS*/  V. 

[  Best  Practices  j 

Exploratory 

Is  this  normal? 
acceptable? 


Questions  to  the  audience: 

■  What  would  be  your  approach? 
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Problem  Definition  -  “Pain” 
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DEFINE 


Maybe  there  are  (too)  many  defects  leaking  from  previous  phases? 


Exploratory 


Yes...  But  we  have  already  tried  code  reviews  and  design 
reviews... 

The  developers  said  they  do  not  work. 


r 


Maybe  your  code 
review  is  inadequate? 


They  said  that  we  find  only  trivial  defects  and  code  style  errors. 
These  can  be  found  automatically  by  code  analysers 


Anyway,  you  agree  that  one  potential  reason  that  your  testing 
process  is  taking  too  long  or  is  expensive  is  the  fact  that  we  have  too 
many  defects  leaking  from  the  previous  phases 

The  only  way  to  know  is  learn  more  about  the  number  of  defects,  the 
type  of  defects  found  during  testing,  how  much  they  cost... 

Do  we  have  this  data? 

--  Yes,  not  all  of  it 
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Problem  Definition  -  “Pain” 
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DEFINE 


Exploratory 


#  Initial  Problem  formulation: 

Low  process  yield  before  QC  (testing).  This  results  in  high  number  of 
defects  that  are  discovered  by  the  QC  team  relatively  to  the  total 
number  of  defects  found  in  process.  The  majority  of  the  projects  (76%) 
find  between  70%  to  100%  of  defects  during  the  QC  phase. 
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Defects  Characterisation 
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Distribution  of  Defect  Impact 


100 


80 


20 


ID 

Percent 
Cum  % 


107875231  67693034  3735545 
56.9  35.7  2.0 

56.9  92.7  94.7 


2100106 

1.1 

95.8 


8024467 

4.2 

100.0 


Distribution  of  Defect  Cause 


100 


80 


40 


20 


MEASURE 


92%  of  defects  found  during  testing  are  either  Ul  or 
Functional 

They  both  affects  the  User  but  may  be  originated  from 
different  sources 

We  can  also  question  the  quality  of  the  defects 
categorisation 
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Elements  of  cost  related  to  Defects 


$ 


systonomy 


MEASURE 


Assess  the  defects  on  multiple  dimensions 


-Low,  M,  H 

-Pairwise  comparison 


-Testing  (common) 

-  Discovery  (test) 

-  Reporting 
-Fixing 

-  re-test 


#>  Rough  Estimate  for  the  business  case 

Effort  use  Median  with  5  estimation  (97%  probability  that  the  median  is  within  the 
min  and  max  of  the  5  values) 


Detailed  estimate 

Calibration  phase 
Estimation  phase 

Experimentation  for  assessing  the  capability  of  the  defect  categorisation 
(measurement) 

Attribute  Agreement  Analysis 
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Root  causes 
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Distribution  of  Defect  Impact 


200000000 


150000000 


100000000 


50000000 


Defect  Impact  U1 

ID  10787|5; 
Percent 
Cum  % 


V 


Functionality  Clarity  Maintenance  Other 
1231  67693034  3735545  2100106  8024467 

56.9  35.7  2.0  1.1  4.2 

56.9  92.7  94.7  95.8  100.0 


Ul  and  Functional  defects 
have  probably  different 
origins 


You  don’t  address  and  you 
don’t  verify  Ul  defects  and 
Functional  defects  the 
same  way. 


Two  different  streams 


ANALYSE 


1 

Alignment 

Code 

2 

Design  consistency 

D 

3 

Field  Corruption 

F 

4 

Functionality 

Ignored  in  Ul 

5 

Naming 

N 

6 

Menu 

M 

7 

Paging 

P 

8 

Translation 

T 

9 

Spelling 

S 

10 

Style 

Y 

11 

Usability 

U 

12 

breadcrumb 

B 

Pareto  Chart  of  Defect  Type 
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XT 

V 

fc 
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60  g 

40  I 
20 


Defect  Type 


*  ^  ^ 


*  *  i 


xj’ 

V  *  •* 


oQ<^  ^ 


a*' 


^  it 


Frequency 
Percent 
Cum  % 


29  22  12  10  7  5  4  3  3  2 

29.3  22.2  12.1  10.1  7.1  5.1  4.0  3.0  3.0  2.0  2.0 

1.3  51.5  63.6  73.7  80.8  85.9  89.9  92.9  96.0  98.0  100.0 


2 

2.0 


29 
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Root  causes  -  problem  focus 


$ 


systonomy 


4-  the  problem  is 
likely  in  the  Ul 
design  process 


ANALYSE 


These  types  of  defects  will  not  be  caught  by  review 
(maybe  visual  inspection) 

Ul  defects  are  not  worth  addressing  by  reviews 


| 

Code 

n 

- > 

Testing 

1 

Defects 
Reaching 
customers 


1-  Initial  problem 
perception 


3-  Maybe  the 
problem  is  with 
code  containment 


cost 

time 

effort 


2- second 
perception 


Locally  Optimise 
testing 


danger 
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Root  causes  -  problem  focus 


$ 


systonomy 


ANALYSE 


Ccfftifiatioitt 


Stills  of  1/1  ncwcnmcr*  - 


Start  working  with  no  lull  graphics  design 
arid/ai  pnp^pp  ar  before  tJient  sigh  off 

Unrealistic  deadlines  ■- 

\ 

k 


j  Distribution  of  resources  across  regrons 


s 


c  . 


.  !S'jilc/df(j!  oynnent  process  on  Us 


i 


Frequent  inter  njptiani  from  ll'I  dtvelapers 


Environment  stability « with 

each  thsok  i-v  codt  changes 


Lj-rk  pf  U]  technical  lead's  ond/cr  wmnrs 
TDlt  to  rflvistv  delimabks  ot  junic-Ts 

Jfi-du-ctio-n  training  foe  5D.  Ul 
Np  SpH  check  for  SD  w^fle  - 
Tech,  (Lerlifkatians  for  U!  dev 


\ 


\ 


i 


Prototype?  and  should  b? 

developed  by  UI  not  graphics  de^gneis 


F 


Tod, 


U]  Psy  iSjndL'ids  &re  ndtfuHy  communicated  tc  graphic* 
to  avoid  developing  Lin-implementable  designs 


|Ut  ifae^  rial  review  grapl~tit;y  design  implimentebjjKy 


hfo  chKkpoinls  op  (ht  output  of  the  UI  before  the 
solution  team  ftrpitrti  i!  lit deFrclS  gcnerntrci  Jt  the  nnd 
are  nut  known  whether  they  are  injected  by  Ul  nr  sol. 


No  chance  tor  U3  developers  lo  review  the  graphic-s-  d«gn  before  customer  sign  off 


When  un-implimentable  graphics  design  is  imposes  by  customer  or  a  thifd  party 


No  ihemfrS  adopTrd  far  Stylfci  (ITS  rtieiritS'l  - 

Using  A^ura  tar  prototyping 


j  No  yer-ipNt  UJ  fiarntwolk  lor  Ul  devtlupcrS  \u  UJS  ; 


No  common  style  sheets  for  al  I  projects 


blueprint  proems  with  Share  pomi  2Q}Q  - 
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Work  Products  &  Transitions 


Role 


Detailed  Process  Map  x  Defects 


Gemba 


systonomy 


Other  factors  influencing  this  step: 
--  Complexity 
--  Size 

--  Number  of  components/widgets 
-  Skills 

V- Tools _ J 


A,  P,  V,  M,  B 

■  ■■  -■  XSL  ■  -»  ■ 


$Ome‘tirne-& 
■  Done  First  i 


o  iTteti  . 


A,  U 


iTifurmal  Cheek  Point  & 
Lli  Cheek  List  Used 


Informal  C 


lcck  Point 


Build 
Script 
+  Manus 


#■  Defects  occur  either  at: 

■  Handover  from  one  level  to 
another  (V) 

■  the  transition  from  one 
tool/environment  to  another 

■  Within  the  same  activity 

#■  Which  type  of  defects  occur 
and  at  what  level  of 
interaction? 


Style  Guide 
i  Design 
Guidelines  for  Ail 
Pages] 


Homo  Page  & 
Sample  Inner 
Page 


Setting  & 
Backup/ 
Resters 
Mechanic  i 


A.  U 


i  W 


Informal  Check  Point 
U1  Chec  k  Li  si  Used 


Testing  an  the  Real  Browsers 


All  Web  Pages 
{HTML,  CSS,  images,  J- 
ScrLpts  S  Other  Scripts]! 


Done  First, 


’  "^1 

|  Adding  Controls. 

&  Link  with  OBs 

BESSi^fiS! 

Function  si  Test 
Cases 


Requirements- 


(RSDf 


Share  Point  Web 
Site  {HTML,  CSS) 


Fp  hi,  T,  V,  D 


Out  Of  The  Box 
Share  Point 
Components 

Dh  F\  Y 


Prototype  Design 
| Menus,  Pages, 
Scenarios) 
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Data  Analysis 


$ 
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ANALYSE 
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Solutions  Identification  and  Evaluation 


$ 


systonomy 


♦  Sources: 

■  Handover 

■  Transition 

■  SP  Technology 

■  DB/Controls 


Solutions 


Ul  Defects 

Importance 

Generartion  of  Layout  from  Graphics 

Generate  code  from  Envt  1  to  Envt  2 

Ul  Reviews 

Define  Ul  Style 

Merging  Layers/Roles  3  and  4 

Naming  convention 

Training/Education 

Dictionaries 

Establish  Checklists 

Merging  Layers/Roles  1  and  2 

Develop  directly  in  target  environment 

Total 

/ 

Field  Corruption 

3.0 

L 

M 

M 

21.0 

02 

Breadcrumb 

2.0 

L 

M 

L 

M 

16.0 

01 

Paging 

2.0 

□t 

\ 

L 

M 

M 

M 

56.0 

04 

Style 

29.0 

□ 

M 

M 

M 

L 

290.0 

22 

Alignment 

22.0 

rez 

tel 

396.0 

30 

Naming 

12.0 

M 

L 

M 

L 

204.0 

16 

Design  Consistency 

10.0 

M 

M 

M 

L 

M 

L 

M 

L 

180.0 

14 

Usability 

7.0 

1  H  i  L 

M 

L 

L 

105.0 

08 

Spelling 

5.0 

L 

M 

20.0 

02 

Menu 

4.0 

L 

L 

L 

12.0 

01 

Translation 

3.0 

L 

L 

M 

15.0 

01 

Total 

251.0 

213.0 

189.0 

138.0 

127.0 

126.0 

63.0 

60.0 

58.0 

47.0 

43.0 

% 

19 

16 

14 

10 

10 

10 

05 

05 

04 

04 

03 

Cost/ Effort 

VH 

VH 

H 

M 

M 

VL 

H 

L 

L 

M 

H 

Risk 

M 

VH 

M 

L 

H 

VL 

L 

L 

L 

H 

H 

IMPROVE 


Solutions 


L>pn«*rTifiii  of  MylkLiT  From  tiiAphin 


Generate  carte  Irom  Eiwt  J  t*  DM  2 


-1-  Mil  liln 


■■■■ 

■III 

'£■  '  ’  M 

Ul  Style 


fvlur^jM^.LdYL'r^/Ktrtrj  i  jn«M 


NiiiriiigiDiiviTiliuii 


MprpiifllavH'i/Rokl  and? 


Develop  tllrectty  In  target  environment 


*M<W. 


fl.O  50.0  ItKMJ  150  0  Z000  Z50.0  iDO.G 
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Solutions  Identification  and  Evaluation 


^  systonomy 

IMPROVE 


Scatterplot  of  Percent  vs  Effort 


Scatterplot  of  Value  vs  Risk 


si 


S2 

S3 


S4 

S6*  S5 

•  • 


-i - 1 - 1 - 1 - 1 - 1 - H 

0  5  10  15  20  25  30 

Risk 
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Process  alterations  and  experiments 


^systonomy 

IMPROVE 


Work  Products  &  Transitions 


Prototypo  Design 
|  Menus,  Pages, 
Scenarios) 


A 


Style  Guide 
(benign 

!  Guidelines  for  Ail 1 
Pages] 

Homo  Page  & 
Sample  Inner 
Page 

— | 

D 

All  Web  Pages 
{HTML,  CSS,  Images.  J- 
ScrLpts  &  Other  Scripts]  1 

Merge  roles  and/or  work  in 
Pair  Design 


automate 


1 


V. 


A 


A,  R,  V,  M1  B 

XSL 


Pul  Of  The  Box 
Share  Point 
Components 


D,  P,  Y 


Sume-time-s- 
-  Done  First  i 


rinhfl  FltSl 

1  Adding  Controls: 

A 

&  Link  with  DBs 

***** 

Ft  Mi  T,  £,  Y,  P 

A.  U 


Informal  Check  Point 
U1  Chock  List  Used 


Share  Point  Web 
Si  te  (HTML.  CSS) 


Fp  N,  T,  Y,  D 


A,  U 


A,  U  I rsform r I  Check  P-pint  & 


nformaf  C  icck  Point 


.  Build 

A  Script 

+  Manuall 


Sotting  & 
Backup/ 
,  Restore 
Maohanrsr 


Testing  on  the  Real  Browsers 


A 

Requirements 

Functional  Test  ' 

(R£D|i 

CafiBt 

Checklists 

Reviews/Usability  Reviews 
Education 

Role  merge  is  difficult 
Due  to  cultural  barriers 


Improvement  associated  to 
all  actions  and  not  to  single 
actions 
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CONTROL:  show  and  validate  improvement 


$ 


systonomy 


♦  A  quick  visual  summary  can  show  the 


Learning  and  adaptation  phase 


controiT^ 


UCL=4./8/ 


1  3  5  7  9  11  13  IS  17  19 


Ul  defects  per  type  per  test  cycle 
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Summary 


systonomy 


#■  The  mother  of  all  Six  Sigma  tools  is  naive  questioning 
#  We  don’t  take  time  to  observe  our  processes 

#■  Studying  defects  provide  lot  of  insights  on  process  improvements  and 
applicable  techniques 

#■  Data  is  A-Political 

■  Test  two  alternatives  and  see  what  the  data  will  say... 

■  Drive  things  by  consensus  (even  if  the  solution  seems  obvious) 

#■  Empirical  software  engineering  is  practical  with  lot  of  pragmatism 

#■  Economics  of  software  engineering  is  a  key  ingredient  to  Empirical  Software 
Engineering 
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Other  key  takeaways...  •  /  :l  n  ':v 

#  Do  not  focus  on  the  tools,  but  rather  on  the  principles 
#■  Do  not  apply  the  method  by  the  books 

#  Do  not  just  Copy  and  Paste  a  practice  or  a  technique 

#■  Do  not  just  dismiss  a  practice  because  someone  has,  understand  why? 

#■  Adopt  the  method  to  your  business,  context  and  environment 
#■  Do  not  follow  the  process  for  the  sake  of  the  process 
#■  Recognise  that  there  is  no  perfect  project 

#■  Recognise  that  human  are  not  perfect...  and  engineers  and  customers  are 
humans 

#■  Recognise  the  importance  of  human  factors...  and  culture 

Never  forget  the  Business 
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ubcontractors  Quality  with 
Cross  Constellations  and 
Multi  Models  Inspiration 

Continues  Process 
Improvement  Initiatives 


+972522946676 


Background 
_he  Challenge 
he  Solution 


Tips  For  You 


Agenda 


Complex  and  large  organizations  or  divisions  that 
run  a  system  /  product  lifecycle  End  to  End,  need 
to  use  more  than  just  ‘one’  CMMI  or  on  quality 
related  standard. 


Our  experience  shows  that  these  organizations  are 
typically  structured  as  matrix  organizations,  with 
functional  teams  or  as  a  complex  of  independent 
smaller  business  units. 


The  Challenge 


This  situation  where  organization  is  running  a 
system  lifecycle  a  matrix  with  internal  or  external 
contractors,  with 

•  With  partial  overall  view  in  interactions  and 
handshakes  between  these  groups  is  introducing 
inefficient  usage  of 

resources, 

expensive  maintenance  of  duplicate  infrastructures 

and  Organizational  Sets  of  Standards  Processes  as  well  as 

assets, 

•  May  result  in  less  quality  and  impacting  the  end 
product  /  system. 


The  Challenge 


This  situation  where  organization  is  running  a 
system  lifecycle  a  matrix  with  internal  or  external 
contractors,  with 

•  separate  process  improvements  on  different  parts  of  the 
system  /  product  lifecycle 


The  Challenge 


This  situation  where  organization  is  running  a 
system  lifecycle  a  matrix  with  internal  or  external 
contractors  =  service  providers,  with 

•  separate  quality  management  systems  and  with 
compliance  to  different  standards  (e.g.  AS9100c)  and 
qualification  (e.g.  MIL-STD  217)  on  different  parts  of 
the  system  /  product  lifecycle 
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The  Case  Study  Organization 
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-900  In-house  Development,  Service 
and  Maintenance  Personal 

-2000  External  Contractors 

Internal  R&D  Team 

Internal  Reliability  and  Performance  Team 
Internal  maintenance  and  support  units 
Internal  manufacturing  and  assembly  units 


Common  Failures  -  1 


ganizational  risk  events  are  predominantly 
managerial,  not  technical. 

Lack  of  defining  business  objectives  in  quantitative  terms  and 
structure 

Inadequate  definition  of  ’Good  Enough’  level 

Inability  to  differentiate  different  business  objectives  and 
success  factors  for  the  different  domains  and  lifecycle  phases 

Inadequate  resource  usage  and  adjustment  to  Plan  and 
Objectives 

Failure  to  identify  and  manage  risks 
Poor  or  mismanaged  service  /  operational  requirements 
Uncontrolled  baselines,  no  configuration  management 
Misunderstood  business  /  operational  needs  and  objectives 


Common  Failures  -  2 


Poor  contractor  acquisition  or  management 
Lack  of  skills,  capability  and  training 

Poor  planning  and  tracking 

•  Value  Stream 

•  Equipment 

•  Resources 

•  Finance 

Poor  /  misuse  of  data  and  measurements 
Inability  to  estimate  accurately 
No  quality  assurance  /  control 
Poor  communications 


The  Operational  Need 


Management  capability  level  from  both  professional  and 
knowledge  level 

Performance  and  reporting  norms 

Self  management  and  self  discipline  maintaining  personal 
professional  and  knowledge  capabilities 

Individual  and  team  discipline 

Cooperation  and  knowledge  and  resource  sharing 

Appropriate  visibility  of  information,  data  and  capabilities 

Quality  of  readiness  and  preparedness  for  performing 
mission 


The  Operational  Need 


Centralized  resource  management  and  appropriate 
utilization  and  usage  of  it 

Multidimensional  management  (future  planning,  unit 
strategy,  short  term  objectives,  the  immediate  objectives) 

Initiating,  developing  and  implementation  management  of 
new  processes  and  technologies 

Balanced  planning  and  deploying  new  processes  and  tools 
improvements  and  new  technologies  in  a  measured  way 
that  will  quantify  the  improvement  vs.  expectations 

Information,  data  and  communication  security 


The  Operational  Need 


ch  person  working  in  the  implementation 
rganization  will  need  to  do  the  following: 

Access  the  processes  descriptions 
Understand  the  lifecycle  at  a  top  level 
Understand  in  detail  of  the  processes  that  he  or  she  performs 

In  addition,  managers  must  do  the  following: 

Understand  the  lifecycle  at  a  top  level 

Understand  the  leadership  change  management  expectations  in  detail 
Understand  how  to  lead  the  unit  using  the  new  processes 

Access  historical  measurement  data  for  all  processes  and  product 
versions  performance 

Support  implementation  of  new  processes  in  their  own  surroundings 
Remove  roadblocks  to  implementation 


m&mm, 


A  Complex  Effects-based  Environment 


Military  Combat  Services  Support 
Challenges  in  the  Battlefield  C4ISR  Systems 


D  Major  Risk? 


el  Systems 


pporting  Systems 

©  ^ 

© 


Extended  Systems 


The  Approach  to  the  Solution 

Concept 

Best  practices  in  the  model  focus  on  activities  for 
providing  quality  services  to  the  customer  and  end 
users 


To  identify  improvement  targets  in  main  lifecycle 
areas  such  as  operations,  information,  governance, 
people  and  organizational  structure,  portfolios, 
project  execution,  and  finance 

Select  processes  that  are  critical  to  the  system 
success  such  as  stakeholder  management, 
technical  interfaces  and  integration 


The  Approach  to  the  Solution 

Concept 

Build  an  action  plan  composed  from  the  following  main 
steps 

•  Organizational  map 

•  Functional  team  and  groups  size  and  role  in  the  lifecycle 

•  Full  lifecycle  map 

•  Setting  improvement  targets 

•  Gap  analysis 

Suggesting  to  the  senior  management  to  address  the 
lifecycle  and  process  (as  a  whole)  as  a  complex  of  crossing 
interfaces  and  to  add  additional  content  to  the  lifecycle 
map  (as  a  layer) 


The  Conceptual  Solution 


Building  on  contingency  theory,  it  outlines  a 
comprehensive  framework  suggesting  a  fit 
between  the  level  of  Mission  interoperability  and 
environmental  as  well  as  internal  contingencies. 

Moving  from  the  current  environment  of  basic 
process  and  way  of  thinking  toward  a  more 
controlled  and  measured  process  to  reduce  the 
overwhelming  amount  of  information  that  build 
decisions 


The  Conceptual  Solution 


We  have  found  that  Maturity  Models  and  practices 
combined  with  some  other  industry  standards  and  methods 
as  a  new  integrated  approach  can  be  used  as  tools  to 
leverage  procedures  to  support  the  lifecycles  and  the 
organizational  business  objectives  and  capability,  readiness 
and  preparedness  to  achieve  improvement  and  excellence. 


*  Information 


People 


Applications 


Business 
rocesses 


p://wiki.  exoplatform.  com/xwiki/bin/download/MainA/VebOS+concept/web_as_a_platform.png?heigh 


JThe  Proposed  Solution  Concept 

Using  the  CMMI-SVC  as  an  overall 
umbrella,  to: 

•  Increase  results  and  effectiveness 

•  Reduce  quality  related  activities  costs  by 
reducing  overlaps  and  choosing  the  appropriate 
parts  only  as  part  of  the  ‘whole’ 

•  Reduce  administration  costs  by  improving  the 
ability  to  manage  the  lifecycle  network 

•  Converged  working  network  helps  businesses 
to  save  procurement  costs  of  infrastructure 


Process  Improvement 
Effort  Objectives 

Group  Target  is  Process  Improvement: 
Increase  Processes  Efficiency 
Increase  Budget  utilization 
Reduce  Cost  of  Poor  Quality 
Increase  Uniformity  in  Processes 
Leading  Standards  to  Compliance  with 
Internal  Quality  Standard 
EFQM 
CMMI  Suite 


Smart  Grid 
ISO  25999 


ACQ  PMs  /  PMO 

•  PMBOK 

•  DoD  5000.01  &  5000.02 

Maintenance  and  Service 

•  MIL-STDs 

•  ISO  14000 

•  OHAS  1 8000 


Supporting  Quality 
Standards  Mapping 


SGMM  Tool  Slides 


'Additional  Standards  Elements 

i  (applied  internally  and  to  contractors) 

Not  Counted 

•  Domain  Specific 
Regulations 

•  LEAN 

•  SOA-MM 


ISO  9001-2008  =  216 
OHSAS  18001  =  132 
L  ISO  27001  =  126 
ISO  27002  =  134 
ISO  14001  =  139 
PMBOK  3rd  =  804 

!OPM3  =  1402 
DoD-AF  V2  =  40 
ISO  20000=  196 

!ITIL  V2.0  =  741 
Six  Sigma  =  148 
MIL-STDs  =  127 
EFQM  =  804 


Additional’  Elements 


Some  Mapping 
Examples 


DEFINE 


ANALYZE 


;1  IMPROVE 


■  CONTROL 

ki'  a 


DAR  RSKM 


PPQA 


STSM 


.  JFC 

-fe 

Management 

Responsibility 


Measurement 
Analysis  & 
Improvement 


Product 

Realization 


Resource 

Management 


Quality 

Management 

System 


DAR  RSKM 


ISO  9000:2008  Correlation  Snapshot 


RP 


SCO 


20  are  covered 


mm 


inancial  Management 

for  /' 
IT  Services  // 


Capacity 

Management 


/  IT  Service 
Continuity  Managemerv 


Availability 

Management 


Service  Level 
Management 


STSM 


ITIL  —  CMMI  Correlation  Snapshot 


Service  Delivery 


SCO 


RSKM 


RP 


DAR 


DAR  RSKM 


The  Service  Desk 


Incident  Management 


Problem  Management 


Configuration 

Management 


STSU 


CMMI  Harmonization 
Process  Tool 


First  Level  Filtering  (PA  Level) 


DEV 

ACQ 

SVC 

r  Project  Planning 
'  Project  Monitoring  and  Control 
:  Process  and  Product  Quality  Assurance 
Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

Project  Planning 

Project  Monitoring  and  Control 

Process  and  Product  Quality  Assurance 

Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

Project  Planning 

Project  Monitoring  and  Control 

Process  and  Product  Quality  Assurance 
Requirements  Management 

Configuration  Management 

Measurement  and  Analysis 

H 

Organizational  Process  Definition  +IPPD 
Organizational  Process  Focus 

Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management +IPPD 
;  Risk  Management 

Organizational  Process  Definition 

Organizational  Process  Focus 

Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Risk  Management 

Organizational  Process  Definition 
Organizational  Process  Focus 
Organizational  Training 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Risk  Management 

■  ■ 

Quantitative  Project  Management 
■  Organizational  Process  Performance 

Quantitative  Project  Management 

Organizational  Process  Performance 

Quantitative  Project  Management 
Organizational  Process  Performance 

■  | 

Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 

Supplier  Agreement  Management 

Supplier  Agreement  Management 

Requirements  Development 

Validation 

Verification 

Acquisition  Requirements  Development 

Acquisition  Validation 

Acquisition  Verification 

Technical  Solution 
Product  Integration 


[Solicitation  and  Supplier  Agreement  Development 
lAgreement  Management 
lAcquisition  Technical  Management 


Capacity  and  Availability  Management 
Incident  Resolution  and  Prevention 
Service  Continuity 
Service  Delivery 
Service  System  Development 
Service  System  Transition 
Strategic  Service  Management 


The  Most  Effective 
Practices  to  Ensure 
Contractors  Qualification 

and  Quality 


Based  on  ~1600  tasks  and  projects  analysis 

and 


Presented  with  practical  usage  and 
implementation  tips 


Achieved  Improvement 


mprovement  vs.  Implementation 

Process  Improvements 


ROI 


5 

o 


GP  2.4 
GP  2.3 


Low 


OT 

SST 


♦  SCON 
SD  IRP 

GP  2.7 

GP  2.5 

GP  2.1 


CM 


m  IPM 

♦  DAR  <  REGJVI 

TS 


PI  STSM 


VAL 


♦  VER 


4  E>RlC 


Implementation  Effort 


High 


Organizational  Benefit 


provement  vs.  Benefit  Add  Value 

U _ 


SST 

GfflDO 


RD 


VAL 


♦  PP 

REQM 

-OT 


♦  DAR 


♦  VER  x  Pl 

A  TS 


CAM 


¥)prpPP 


QPM 


OPF 


M&A 


STSM 
■  SD 
♦  SCON 
a  RSKM 
■  I  PM 


IRP 


Low 


Achieved  Improvement 


High 


Some  of  Our  Suggestions 


Cracking  the  Code 
for  Improving  the  Productivity  of 

Knowledge  Workers 


Peter  Voldby  Petersen 
Dvp(d).callis.  dk 

www.callis.dk 
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Force  multiplication! 


V  Force  multiplication  refers  to  a  combination  of 
attributes  or  advantages  which  makes  a  given  force 
more  effective  than  another  force  of  comparable  size. 


V  A  force  multiplier  refers  to  a  factor  that  dramatically 
increases  (hence  "multiplies")  the  effectiveness  a  group. 


Some  common  force 


Morale 
Technology 
Geographical  features 
Weather 


What  are  the  ’’force  multipliers” 
when  talking  about 
improving  the  productivity 
of  knowledge  workers? 


Recruitment  through  diplc 
Training  and  experience 
Fearsome  reputation 
Deception 
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The  idea  with  this  presentation 

V  The  idea  is  to  convey  some  of  our 
ideas  and  thoughts  about  aspects  are 
important  related  to  the  technical 
infrastructure  supporting  process 
modeling,  authoring,  tailoring,  use,  and 
improvement  in  knowledge  worker 
environments 


V  Hopefully,  you  can  use  this  as 

inspiration  when  building,  improving  or 
evaluating  process  infrastructures  to 
make  them  true  force  multipliers  for 
your  organization 


Copyright  ©  2007  - 11,  Callis 


CALLIS 


(Simplified)  Concept  of  Operation 


Process 

owners 


Document,  X-ref  .Checks, 
Overviews,  Change  Log,  Publish 


Organizational 
Process  Set(s) 


>1 

© 

H  > 

<> 

AE 

Define 


Project’s 
Defined  Process 


Suggest 

improvement 


Tailor 


Project  A 
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Work  Products 

call's 


Process 

owners 


The  Force  Multipliers 


Dc 

On 


Automate 

’’build  &  deployment” 


Organizational 
Process  Set(s) 


>1 

© 

H  > 

<o> 

AE 

7" 


Tailnr 


For  each  of  these 

•  The  often  seen  situation 

•  Consequences  for 
knowledge  worker 
productivity 

•  An  alternative  approach 
-  Examples 


Proj 

Defi 


Change  to 

’’Continuous  Tailoring” 


► 


AE 


Harvest  (pull)  the 
’’proven  in  practice” 
_ improvements _ 


Project  A 


Work  Products 

call's 
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Automate  ’’build  and  deployment” 


Often  seen  practice  Process 

V  Lots  of  manual  work  and  owners 

schedule  required  to  from 
change  to  release 

*9  Resistance  to  change  in 
process  group 

9  Bi-yearly  releases  of  process 
sets 

Consequences  for  Know.  Workers 

*7  Non-optimal  presentation  of 

process  sets  (process  group  starts 
to  think  of  ’’ease  of  maintaining”  insted 
of  ’’ease  of  use”) 

9  Not  updated  to  “real  practice” 


Organizational  Tai|or 
Process  Set(s) 


9  Lower  quality  of  documentation 
9  Slows  down  the  “learning  loop” 
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Automate  ’’build  and  deployment” 


An  alternative  apprach 

V  Strive  towards  ’’Continuous 
build  &  deployment”  setup 

*7  ”Do  things  which  are  hard 
often” 

*7  Drive  down  the  effort  and 
schedule  required  to  build  & 
publish 

Example 

d  Cj  Processes 

apcalfe  Engineering  Process 

Callis  Engineering  Process  (3.7  -} 


Process 

owners 


From  change  to  action! 
Effort? 
Schedue? 


✓ 

Author  Process 

Sif 

Publish  Process 

Organizational  Taj|or 
Process  Set(s) 


B 

<> 

AE 

Project’s 
Defined  Process 


AT4 
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...in  less  than  5  min... 
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Change  to  ’’Continuous  Tailoring” 


Often  seen  practice 

9  Tailoring  by  copy-paste  process 
descriptions 

V  Tailoring  in  a  file  separate  from 
process  set 

V  Tools  where  tailoring  is  practically 
impossible 

•  Yes,  we  can  do  that,  but  it  requires  an 
army  of  consultants  and  a  phd  in 
process  modelling 

Consequences  for  Know.  Workers 

V  Doesn’t  capture  the  real  process 

9  Tailoring  turns  into  a  “formal  thing” 

V  Limited  ’’connection”  between 
documented  process  and  real 
process  -  the  WIKIs  take  over... 

9  Improvements  are  not  based  on 

c’p^&mtPhffprocesses 


Organizational 
Process  Set(s) 


CALUS 


Change  to  ’’Continuous  Tailoring” 


An  alternative  apprach 

V  Combine  structured  processes 
with  wiki  approach  - 
’’continious  tailoring” 

*7  Make  it  easy  to  add 

operational  comments  directly 
into  the  process  set 

Example 


Project’s 
Defined  Process 


Tailor 


Use 


Project  A 
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Change  to  ’’Continuous  Tailoring” 


2 

Reouirements 

Manager 

Document  needs  and  expectations 

The  needs  and  expectations  should  be  documented  in  the  Requirements 

Management  Plan. 

3 

Requirements 

Manager 

Decide  which  Requirements  Management  System  to  use  _ _ 

If  no  special  technical  or  customer  related  needs  of  a  special 

Reouirements  Management  System,  the  organizational  standard  system  must  be 
used. 

4 

Reouirements 

Manager 

Prepare  specification  of  project'specific  setup  of  Requirements  Management 
System  / 

Document  needs  and  expectations 

The  needs  and  expectations  should  be  documented 
Management  Plan. 

Decide  which  Requirements  Management  System  t 

If  no  special  technical  or  customer  related  needs  mar 
Requirements  Management  System,  the  organizatior 
used. 


We  need  to  use  RM-Pro  in  our  project  -  this  is  a  part  of  the  contract  with  the  custJ 

/ 


Prepare  specifi 
System 


Requirements 

Manager 


Requirements 

Manager 


Requirements 

Manager 


Document  r 


The 

Man 


needs  ; 
aaenier 


Low  ceremony 
Empowers  people 
Practical  project  learning 


Decide  whi< 

If  no  specia 
Requirement 

us  2d. 

y 

We  need  to  use  RM-Pro  in  our  project  -  this  is  a  part  of  the  contract  with  the 
customer 


Project  [ 


Prepare  specification  of  project  specific  setup  of  Requirements  Management 
System 


13:28  by  pvp 

m 
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Learning  Across  Projects 


Short  circuit  knowledge  generation 
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Harvest  (pull)  the  ”proven-in-practive” 

improvements 


Often  seen  practice 

V  Employee  ’’push”  improvement 
suggestions  to  a  queue 

*7  Hard  to  see  the  "quality”  of  the 
improvement  suggestion 

•  Theory  or  ’’proven  in  practice” 

^  Improvement  suggestions 
queue  up,  action  slows  down 

^  Learning  loop  does  not  work 

Consequences  for  Know.  Workers 

^  “The  EPG  /  process  owners 
don’t  do  anything” 

S 7  Outdated  process  descriptions 

^  Slow/limited  build-up  of 
intellectual  capital 


Process 

owners 


Project  A 
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Harvest  (pull)  the  ”proven-in-practive” 

improvements 

An  alternative  apprach 

V  ’’Pull”  the  ”proven-in-practice” 
operational  tailorings 

<7  Do  this  in  the  ’’structure  of  the 
process”  as  this  makes  it  easy 
to  compare  tailorings  across 
projects 


Process  Set 


C3 

<> 

AE 

Example 


Process 

owners 


Project  A  Project  B 
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Harvest  (pull)  the  ”proven-in-practive” 

imnrnvfi  merits 


&  Establish  Requirements  Management  System 

Summary 

Coordinate  s 


Main  Desc 

This  task  de: 

Entry  Crite 

Check  this  oi 

Steps 


Rol< 


Oversee  tailorings 
across  multiple  projects 


9 


*  Requirements  b 

Accountable 

*  Project  Manage 

Consulted 
■  Test  Manager 


-  X 


Modified 

User 

Project 

Comment  Element 

2009.11.031... 

pvp 

Project  B 

As  we  are  doing  internal  product  development  with  no  access  to  end  users,  our  product  manager 

2009.11.031... 

pvp 

Project  B 

In  Project  B  we  need  to  use  this  system... 

2009.11.031... 

pvp 

Project  E 

We  need  to  use  RM-Pro  in  our  project  -  this  is  a  part  of  the  contract  with  the  customer 

v 

< 

s_i 

s± 

ice 
s  h 


Requirements 

Manager 


Document  needs  and  expectations 

The  needs  and  expectations  should  be  documented  in  the  Rep^ 


9 


As  we  are  doing  internal  product  development  with  no  acces 
manager  will... 


Requirements 

Manager 


Decide  which  Requirements  Management  System  to  use 

If  no  special  technical  or  customer  related  needs  mandates  th^  use  or 
Requirements  Management  System,  the  organizational  standard  sys 


In  Project  B  we  need  to  use  this  system... 


See  the  tailoring  in 
the  process  description 


sed. 


Project  E  |  2009,11,03  14:04  by  pv 


We  need  to  use  RM-Pro  in  our  project  -  this  is  a  part  of  the  contract  with  the  customer 


Project  E  |  2009.11.03  14:03  by  pvf 


F 

5 


The  new  ’’Concept  of  Operation” 


Automated  ’’model, 
author,  build,  and 
publish”  mechanism^ 


Automate 
’’build  &  deployment” 


J 


Process  sets 


Process 

owners 


Pull  mechanism  build  on 
practical  experiences  and  enables 
organizational  learning 
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Harvest  (pull)  the 
’’proven  in  practice” 
improvements 


Project  A 


Change  to 

’’Continuous  Tailoring” 


Low  ceremony, 
tailoring  mechanism, 
empowers  people 
and  enables  local 
learning  and 
improvement 


call's 


Thank  you! 


V  Please  contact  pvp@callis.dk  for  demos  and  trials 

•  Or  just  to  discuss  the  concepts  © 
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Backup  slides  from  here 
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Stuff  you  need  to  address... 

*7  Well  defined  process  architecture  supporting  re-useable 
process  content  /  method  libraries 

*7  Automated  consistency  checking 

*7  Generate  and/or  integrate  graphics  (MS  Visio  Addin) 

*7  Multi-model  compliance  mapping  /  cross  referencing 

*7  Multiple  views,  multiple  variants,  multiple  output  formats 

*7  Activation  -  e.g.  instantiating  roles  and  best  practices 
(MS  Sharepoint  integration) 

V  Lightweight  tailoring  /  operationalization 
*7  Understand  tailoring  across  projects 

V  Profiling  use  of  processes 

*7  Automated  compare  (diff)  process  definitions  /  change 
list  Sf 

CALLIS 
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Supporting  knowledge  workers  is 
supporting  the  lifecycle  of  processes! 

Organize,  structure,  and  Manage  compliance  to 

author  process  sets  standards  and  requirements 


Compare 
process  sets 


Understand  tailoring 
across  projects 


Consistency  and  uniformity 
(in  text  and  graphics) 


Automated  publishing 
of  multiple  versions 


Effective  access  to 
process  sets  and  assets 


Lightweight  and  practical 
tailoring 


Integrate  best  practices 


Activate  the 
process,  e.g. 
artifact  and  role 
instantiation 
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Profile  use  of 
process  sets 
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Process  content 


From  Process  Modelling  to  Process  Execution 


Packages 


Process  Sets 


Scrum 
web  project 


ject^/ 


o 


WIP 


Waterfall, 
fixed  reqs 
project 


2.1 


Author 

•  Cross  refs 

•  Consistency 

•  Generated  pics 

•  Diff.  Process  sets 


Publish  ^ 

•  To  Callis  Performer 

•  Static  website 

•  PDF  “l 


C 

o 

"(/) 

s 

.Q 

CO 


Published 
Process  Sets 


Services 


\  . 

1  -  -  ^ 


Project  A 
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Tailoring  Sets 

(Operational  comments) 


CALLIS 


BaiGuo  ML4  Journey 


Tim  Kasse 

Kasse  Initiatives  LLC 
+1  817  576  3142  USA 

+45  (0)  72  19  42  18  Europe 

+1  303  275  3285  NREL 


Edmond  Sung 
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How  a  focus  on  high  maturity 
CMMI-based  process 
improvement  can  add  value 
to  the  organization  even 
when  there  are  only  limited 
resources 
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^  BAI  GUO  Background  and  Culture 

^  BAI  GUO  Process  Improvement  Methodology 

GQ(I)M  method  used  to  align  business  goals  to 
indicators  and  metrics 

^  Data  Analysis  Interpretation 

^  Developing  and  Evolving  the  Prediction  Model 

^  Project  Managers  and  Quantitative  Project  Management 

^  QPM  Story 

^  Summary  of  Six  Sigma  Techniques  Used 
^  Lessons  Learned 

®  Capability  Maturity  Model,  Capability  Maturity  Modeling,  and  CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  &  Trademark 
Office. 
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BAI  GUO  Background 
and  Culture 
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(8  $  1  J?  J#  A  S  8 
DOING  BETTER  TO  BE  THE  BEST 
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^BaiGuo  Info-Tech  Co.  Ltd  is  a  high-tech 
enterprise  professional  on  automation  of  health 
care  in  Shanghai 

<^30%  of  leading  staff  in  R&D  department  have 
been  working  on  medicine  information  industry 
and  long-distance  info  processing  industry  for 
more  than  10  years  in  America 

^International  advanced  CMMI  quality 
management  model  was  chosen  to  be  applied 
to  assure  high  quality  products  and  services 
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Senior  Management  and  other  key  members  of  the 
organization  had  a  CMMI-based  process  improvement 
background  from  other  companies  before  they  formed  or 
joined  BAI  GUO 

A  culture  of  “sharing”  and  supporting  colleagues  is  pervasive 
and  supported  by  Senior  Management 


Peer  Reviews  are  a  key  process  component  of  all  projects 

While  the  “peer”  part  of  the  reviews  is  honored  anyone  who  has  the 
necessary  expertise,  including  the  senior  management  participates  in 
the  peer  reviews  and  truly  acts  as  a  “peer”  during  those  sessions. 

<$>  The  focus  of  these  peer  reviews  is  to  improve  the  product  and 
processes  that  were  followed  to  create  that  product 

Project  Leaders  are  not  only  asked  to  manage  their  projects 
but  to  look  at  their  project  and  other  projects  from  an 
organizational  point  of  view 
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CMMI  ML3  thinking  and  actions  are  considered  the  minimum 
required  for  BAI  GUO  to  control  costs  and  manage  risks 

High  Maturity  is  seen  to  be  the  necessary  component  to 
increase  the  control  and  accuracy  of  Management  decision 
making 

<$  All  Project  Managers  are  aware  of  and  can  share  /  teach  the 
Organization’s  Set  of  Standard  Processes  to  other  managers 
and  employees 

EPG  Lead  not  only  promotes  the  Organization’s  Set  of 
Standard  Processes  but  also  serves  as  BAI  GUO’s 
Measurement  Lead 
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^How  ready  is  the  organization  for  starting  the  ML4 
journey? 

^On-site  check,  using  a  predefined  HM  checklist, 
before  taking  up  the  project: 

Meet  the  senior  management,  EPG,  project  leaders,  and 
their  measurement  group  (if  it  exists) 

Who  will  analyze  the  data?  Experience? 

Capability  of  the  project  leaders  to  interpret  the  analysis 
results  for  managing  their  projects 

Questions  covering: 

-  Vision,  business  objectives 

-  Process:  solid  ML3  foundation  ? 

-  Measurement  repository,  measurement  system,  tools  ? 

-  Stable  processes? 

-  PPB , PPM 
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^Improve  customer  satisfaction 
^Obtain  more  repeat  orders. 

^Increase  company  size  to  100  in  3  years,  and 
<S>  IPO  in  2015. 
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<$>  Improve  the  Customer  Satisfaction  by: 

^Improving  schedule  performance  (reduce  delivery 
delays) 

^Improving  the  quality  of  the  delivered  product 

^Improve  the  development  work  by: 
^Controlling  or  reducing  project  costs 
^Improving  the  productivity 
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Pr< 


Project:  what  are 
leading  in-process 
indicators  of 
success?  Where 
are  improvements 
needs? 

_ 


Business  Objectives 


Customer 

satisfaction 


Reduce 
schedule  dela 


Y's 

Success  indicators, 
Management  indicators 


I 


Improve 

productivity 


Productivity  / 
schedule  data: 
Plot,  plot,  plot 
Trends 
Distributions 
Control  charts 
Scatter  plots 


y's  and  x's 

Analysis  indicators, 
progress  indicators 

Is-  4 


1 


Reduce 

defects/  rework 


Analysis  indicators, 
progress  indicators 


i 


Defect  data: 
Plot,  plot,  plot 
Trends 
Distributions 
Control  charts 
Scatter  plots 
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^Derived  quantitative  objectives: 


By  12/31/2010  our  organization  will  improve 

Productivity  (FP/man-hr) 

from  today's  performance  baseline  of: 

Mean  =  0.26  Std.  Dev  =0.08 


to  a  new  performance  baseline  of: 
Mean  =  0.28  Std.  Dev  =0.06 
with  at  least  95%  of  confidence, 
without  sacrificing  the  product  quality: 


©  2011  Processis  LTD  &  Kasse  Initiatives,  LLC  Version  2.3 


BAI  GUO  ML  4  Journey  - 14 


Quantitative  Objectives  ■  2 


^Derived  quantitative  objectives: 
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By  12/31/2010  our  organization  will  improve  Defect 
Density  (Defects/FP) 

from  today's  performance  baseline  of: 

Mean  =  0.46  Std.  Dev  =0.2 


to  a  new  performance  baseline  of: 

Mean  =  0.43  Std.  Dev  =0.18 

with  at  least  95%  of  confidence. 

Project  delivery  time  is  closely  related  to  the  staff 
productivity  and  the  amount  of  defects  /  rework  (quality). 
In  Baiguo  ,  it  is  believed  that  if  productivity  and  quality 
are  in  control,  the  delivery  is  also  in  control. 
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Problem  and  goal  statement  (Y) 


BAI  GUO  Process 
Improvement 
Methodology 
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r 


Clarify 

Business  Goals 
Process  maps 
Prioritize  issues 


•Define 

measures 


•Discovery:  plots,  paretos,  histograms, 
distributions,  fishbone 
•Collect  &  verify  ’Understanding:  root  cause,  critical  factors 
measures  ‘Performance:  Process  stable? 

Process  capable? 

•Corrective  actions  to  Improve  Process 
Y=f(method,  review  rate,  experience . ) 
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^Duration  of  an  improvement  cycle  depends  on: 

♦  Project  cycle  /  duration 

♦  Number  of  projects  per  year 

♦  Quality  of  project  measurement  data 

♦  How  long  it  takes  to  deploy  a  change 

^Each  improvement  cycle  is  around  3  months  in 
BaiGuo 

^Key  points  to  consider  in  preparing  for  appraisal: 

♦  Any  statistics  specialist  in  the  team  to  analyze  the  data 
(Project  leaders  /  EPG  should  be  able  to  INTERPRE  the 
data ) 

♦  For  the  HM  PA,  evidences  should  be  prepared  to  the 
subpractice  level 

♦  Use  a  Q&A  at  SP/GP  level  to  set  the  appraisal  expectation 
to  the  organization 
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-  an  overview 
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Define 

Measure 

Analyze 

Improve 

Control 

Voice  of  Customer 

Metrics:  defects, 
project  mgmt 

Regression 

Design  of 

Experiments 

Statistical  Controls 

Voice  of  Business 

Data  Collection 

Methods 

Cause  &  Effect 

Modeling 

Control  Charts 

Project  Charter 

Sampling 

Techniques 

Diagrams ,  Matrix 

Eliminate  waste 

Time  Series 

methods 

Kano  model 

Measurement 

System  Evaluation 

Failure  Model 

Effects  Analysis 

FMEA 

Robust  Design 

Other  control  tools 

QFD 

Quality  of  Data 

7  Basic  Tools 

Kaizan 

Sustain 

improvement 

Process  Flow  Map 

"Management  by 
Fact" 

Hypothesis  Tests 

Root  Cause  Analysis 

Decision  &  Risk 
Analysis 

Procedural 

adherence 

Reliability  Analysis 
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Measurement 
and  Analysis 
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Training  Consulting  &  Solutions 
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^  Descriptive  statistics  are  provided  to  show  mean  and 
standard  deviation  for  Productivity  and  Total  Defects 


^Statistical  techniques  used  through  MiniTab  and  Crystal 
Ball  include: 


^Scatter  Plots  to  determine  correlation  between 
Requirements  elicitation  and  size  in  Function  Points  and 
Design  and  size  in  Function  Points  and  CUT  and  size 

^Regression  Analysis  was  used  for  Requirements 
elicitation  and  size,  Design,  and  Code  and  Unit  Test 

^Analysis  of  Variance  was  run  to  determine  statistical 
significance 

^Stepwise  regression  was  run  for  Requirements  Elicitation, 
Design,  and  CUT  vs.  Size  FP  vs.  Team  Experience  vs. 
Requirements  Volatility 
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^  A  linear  regression  relationship  was  able  to  be  built 
between  Requirements  elicitation  effort  and  the  size: 

^>A  similar  regression  relationship  was  built  for  Coding  and 


Size 


^>For  the  design  effort  it  was  found  that  Team  Experience  is 
also  a  factor  in  addition  to  size  (in  FP)  for  predicting  the 
design  effort. 

^  Scatterplots,  Regression  Analysis  and  Stepwise 
regression  were  also  applied  to  Reviews 

♦  Regression  Analysis:  Prep  versus  Reviewed  Size 

♦  Regression  Analysis:  Conduct  Review  versus  Reviewed 


Size 


Boxplots  and  ANOVA  were  applied  to  Preparation  Time 
+  Conduct  Review  time  versus  Review  Type 
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Regression  analysis 
-  an  example 
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Requirement 


Design 


Code/UT 


t 


i 


1.  Relationship  of  Requirement,  Design, 
Coding  to  Size  (FP)  ? 

2.  Regression  analysis: 


System  Test 


Design  [Y]  =  24.8  +  0.224  x  Size  (FP) 
P=0.008  significant 


3.  Effect  of  independent  factors,  such  as  team  Design  =  f(team  experience,  size) 
experience,  reuse,  requirement  volatility  ?  R2  (adj.)  =  92%  ,  P=0.007  significant 
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Process  Map  and  rProcessi$ 

Measures 
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Performance 
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Organizational  Process 
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<$>  Establish  Quality  and  Process-Performance 
Objectives 

Quantitative  Measurement  Objectives  for  Quality  and 
Process  Performance  are  documented  in  BAI  GUO 
Organization  Strategy  document 

SMART  Objectives  were  used  to  define  the  Quantitative 
Measurement  Objectives 

<§>  BAI  GUO  is  in  the  Medical  Hospital  Industry  and  has 
built  IT  systems  to  transfer  patient  records  from 
remote  farming  areas  to  nearest  Hospitals  and  from 
Hospitals  to  Hospitals  in  bigger  cities.  Time  to  market 
and  quality  are  crucial. 

<$>GQM  approach  is  used  to  determine  the  measures 
for  each  objective 
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Organizational  Process 
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The  definitions  of  all  the  required  measures  are  documented 
in  the  metrics  definition  guideline  -  BAI  GUO  Metrics 
Definition  Guideline 

Name  of  metric 
<§>  Business  objective 
Unit  of  measure 
<$>  Formula 
Source  of  data 
<§>  Frequency  of  collection 
Where  the  data  is  stored 
Owner  of  the  data 

Tool  used  to  collect  basic  measures  (automatic  collection  for  basic 
measures) 

Collecting  requirements  to  ensure  data  validation 
<§>  Recommended  statistical  technique 
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Table  of  Basic  measures  with  their  units  of 
measure  has  been  created  and  is  part  of  the 
Metrics  Definition  Guideline 

^Significant  time  was  put  in  to  define  what  basic  measures 
were  needed  along  with  their  units  of  measure 
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^Select  Processes  -  Approach  taken  in  BAI  GUO 

#  Looked  at  all  the  available  historic  data,  by  project  phase 

<^Tried  to  validate  the  data  by  scatter  plot,  and  calculate  the 
summary  statistics  (mean,  standard  deviation)  to  see  if 
the  data  were  reasonable 


#  Correlation  was  used  to  see  what  factors  affected  the 
effort  and  build  the  regression  relationship  between  the 
outcomes  and  the  factors 

<§>  Steps  were  repeated  by  decomposing  the  subprocesses. 
Again  correlation  was  used  to  see  what  factors  affected 
the  effort  and  the  regression  relationship  was  built 

^Preliminary  findings  were  validated  with  the  consumers 
(Project  Leaders)  to  ensure  the  model  and  analysis  made 
sense 
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<$>  Total  Defects  Segmentation 

^Opportunities  for  Improvement  were  identified 
^Improve  development  work 
^Improve  productivity 


Organizational  Process  £ 
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^Reduce  rework  and  Cost  of  Quality  to  improve 
overall  productivity  (Total  Rework  and  rework  per 
defect) 

^Defects  from  each  lifecycle  phase  were  examined 

^Defect  Data  was  collected  from  multiple  small 
projects 
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<$>  Defects  were  broken  down  by  A,  B,  and  C 
defect  categories  with  full  explanation  behind 
them  in  the  Testing  Process  document 

<^A  -  Serious 

<§>  B  -  Major 

<^C  -  Minor 

^Systems  Testing  produced  the  most  defects  -> 
Upstream  activities  were  examined 

<$>  Pareto  Charts  of  A,  B,  and  C  categories  of 

defects  that  contributed  to  rework  were  developed 
by  lifecycle  phase 
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^  FMEA  was  conducted  to  further  categorize  the  CAT  B 
defects  per  lifecycle  phase  and  reduce  the  defects 
coming  into  and  going  out  of  Systems  Test 

<$>  Invite  all  project  leaders,  and  EPG;  Delphi  Method  were  used 

<§>  FMEA  Table  included: 

Potential  Failure  Mode 
Potential  Failure  Effects 
Severity  (1-10) 

Potential  causes 
%  of  Occurrence  (1-10) 

Current  Control 
%  of  detection  (1-10) 

RPN  =  Severity  x  Occurrence  x  Detection 
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Ste  p 

Potential  Potent 

Failure  SEV  ial  OCC  Current 

Potential  Failure  Mode  (from  checklist)  Effects  (1-10)  Cause  (1-10)  Controls 

DEI 

■:i-io) 

Actions  Responsibi  Action 

R  FN  Re c o m m e n d e cl  1  ity  T ake n 

Require¬ 

ment 

Co ntr ad i cti n g/  r e  p e  ate  d  r e  quire  me nts 

8 

7 

4 

224 

N  o  n-f u  n  cti  o  n  al  r  e  q  u  i  r  e  m  e  nts  n  ot  clear 

8 

8 

3 

192 

Others 

6 

8 

3 

144 

Design 

Incomplete,  not  testable 

7 

>5 

4 

163 

D  e  si  gn  n  ot  tr  ac  e  ab  1  e  to  r  e  q  u  i  r  e  m  e  nt 

5 

7 

4 

140 

Not  match  interface  spec 

8 

'? 

3 

144 

Others 

8 

4 

4 

123 

i  Coding 

M o d u  1  e  too  c  o m p  1  i  c ate  d  /  c o in  pies: 

6 

5 

6 

130 

Did  not  avoid  floating  point  operation 

7 

5 

5 

175 

Loop,  branch,  logic  not  correct 

7 

5 

6 

210 

No  checkf or  input  parameters 

7 

5 

6 

252 

Others 

6 

7 

5 

210 

Purpose 


The  purpose  is  to  reduce  the  rework  effort  in  project,  to  find  out  the  defect  source  from  review 
c h e  ■: kl i st  wh ich  i m p acts  r e w o r k  e ff o rt  m o st. 

Here  lists  the  checklist  items  which  impact  the  defect  most. 
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^  Highest  priority  areas  were  examined  as  input  and  a 
Cost-Tradeoff  Analysis  was  conducted 

^  Defects  coming  out  of  Coding  were  focused  on  first  as 
the  most  practical  and  cost  effective  area  to  concentrate 
on 


Organizational  Process 
Performance  Highlights  -  8 
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<$>  Upgraded  Code  Review  guidelines  were  developed  with 
enhanced  guidelines  which  emphasized  lower  level  categories 
from  Coding 

<$>  If  results  from  the  focus  on  Coding  were  not 
sufficient,  the  next  alternative  solution  was 
discussed  with  Senior  Management  to  determine 
feasibility 
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Organizational  Process 
Performance  Highlights  -  9 


Training  Consulting  &  Solutions 


<$>Subprocess  Selection  Guideline 

<$>  Developed  to  support  the  traceability  of 
subprocesses  back  to  Business  Objectives  and  vice 
versa 

EBusiness  Goal 
<^Goal  type 

^Stakeholder  Perspective 
^Typical  Questions 
^Metrics 

^Impacting  Sub-processes 
^Sub-process  control  measure 
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Performance  Highlights  - 1 0 
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^Statistical  Control  (Stable  Processes) 

^XmR  control  charts  are  used  for  Requirements  and 
Coding  to  ensure  productivity  is  statistically 
managed 

^EXAMPLE:  A  spike  in  productivity  was  noticed. 
Causal  analysis  was  conducted.  It  was  determined 
that  there  was  a  high  use  of  “re-use”  at  that  point. 
However,  it  was  decided  that  this  was  an  anomaly 
and  the  control  limits  were  not  changed 
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Control  chart  ■  example 


X  Chart  for  Code  productivity 


Code  productivity 
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Process 

Capability 


Cpm 


Stable  Capable  3.25  2.11 
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Organizational  Process 
Performance  Highlights  - 1 1 


Processis 

Training  Consulting  &  Solutions 


<$>  Process  Performance  Baselines  (PPBs) 

^The  organization’s  Process  Performance  Baselines 
(PPBs)  were  derived  through  analyzing  the 
distribution  of  the  data  to  establish  the  central 
tendency  and  dispersion  (sigma)  to  characterize  the 
expected  performance  and  variation  for  the 
se  ected  processes  or  subprocesses. 

<^EPG  provides  templates  to  Project  Teams  and  provides 
training  on  their  use 

•^Project  Teams  collect  required  data  and  give  it  to  the 
EPG  -  automated 

•^Training  was  also  provided  on  how  to  interpret  the  Excel 
or  Minitab  analysis  results  based  on  historic  data 
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Organizational  Process 
Performance  Highlights  - 12 
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^  The  PPB  and  related  measurements  are  reviewed 
regularly  in  the  EPG  meetings  (by  the  EPG  group  and 
the  PMs),  in  the  project  meetings  (by  PMs  and  project 
team  members),  and  the  senior  management  meetings 
(by  GM,  EPG,  QA,  PMs). 

^  PPB  Summary  Report 
Processes 
<^Subprocesses 
^XmR  Charts  for  productivity 

^  PPBs  are  updated  every  six  months  -  ANOVA  is  used 
to  determine  statistical  significance  of  productivity 
change 
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Organizational  Process  £ 

Performance  Highlights  - 1 3 


Processis 

Training  Consulting  &  Solutions 


<$>  Process  Performance  Models  (PPMs) 

^ There  is  a  Crystal  Ball  predictive  model  for 
productivity  and  quality. 

^Besides  the  crystal  ball,  there  are  also  other  PPMs 
(e.g.  linear  regression)  for  Project  Leaders  to 
reference  to. 

<^E.g.  relating  the  upstream  process  factors  to  predict 
the  activities  downstream  in  the  Minitab. 
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rystal  Ball  prediction 
model  -  an  example 
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Integrated  Project 
Management 
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Integrated  Project 

Management 

Highlights 
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^Project  Closure  meeting  is  held  at  end  of  each 
project  to  capture 

^Project  size  and  effort  tracking  results 
^Actual  work  effort  for  each  phase 
^Deviation  of  effort/schedule  for  every  milestone 
^Measurements  to  update  PPBs 


^Sent  to  EPG  Lead  for  consideration  to  be  included  in 
the  Organization’s  Measurement  Repository  or 
Process  Asset  Library 
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Project  Monitoring  and 
Control  -  Highlights 


<$>  Actuals  to  estimates  tracked  include: 


^Size,  Effort,  Risks,  Stakeholder  Involvement,  Resources, 
Quality,  Non-compliances 

<$>  Schedule  Variance  is  calculated  and  acts  as  a 
threshold  for  corrective  action 


Processis 

Training  Consulting  &  Solutions 


<$>  Cost  Variance  is  calculated  and  acts  as  a  threshold  for 
corrective  action 

XmR  Control  Charts  are  used  to  ensure  process 
stability  -  Special  Causes  of  Variation  are  analyzed 
and  corrected 

At  the  end  of  each  Milestone,  Crystal  Ball  is  run  using 
last  phase’s  data  to  predict  next  phase  results  (Interim 
targets)  and  all  subsequent  phases 
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Quantitative 

Project 

Management 
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Quantitative  Project 
Management  Highlights 


Training  Consulting  S  Solutions 


^Projects  make  use  of  Organizational  selection  of 
processes  and  subprocesses  created  from  OPP 

^Project  Managers  conduct  an  analysis  of  data  in  the 
PPB  to  determine  which  subprocesses  will 
contribute  most  to  meeting  the  measurement 
objectives  of  the  project.  These  are  placed  into  the 
Project  Quantitative  Management  Plan 

<$> Crystal  Ball  model  was  also  rerun  after  each  phase 
(e.g.  req  ph)  milestone  to  predict  the  outcome  from 
the  downstream  processes. 
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Quantitative  Project 
Management  Highlights  ■  2 


Training  Consulting  &  Solutions 


■ 


^ Subprocesses  with  the  most  variation  were 
selected  -  Requirements,  and  Coding+UT. 

mother  subprocesses  are  less  affected,  and  have  less 
variation. 

^>E.g.  defects  at  the  requirement  phase  will  affect  the 
defects  and  rework  in  the  testing  phase. 

^Similar  to  OPP,  analysis  was  carried  out  on  various 
factors  to  determine  the  critical  subprocesses 

^XmR  charts  were  used  on  past  project  data  to  show 
that  the  productivity  of  those  subproceses  were 
stable 
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Quantitative  Project 
Management  Highlights  ■  3 


Training  Consulting  &  Solutions 


^  At  the  end  of  each  phase  the  control  chart  was 
used  to  show  if  the  selected  subprocess  are 
meeting  the  specification 

^Cpk  was  calculated  to  see  that  the  degree  of 
meeting  the  customer  specification  limits  are  met 

^Crystal  Ball  was  used  to  recalculate  confidence 
level  for  to  predict  meeting  the  overall  project  goals 
at  the  end  of  each  phase 
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Quantitative  Project 
Management  Highlights  ■  4 


Training  Consulting  &  Solutions 


I 


^The  definitions  of  the  common  measures,  units  are 
found  in  the  organizational  guideline  BG  Metrics 
Definition  Guideline 

^The  USL  and  LSL  of  the  objectives  of  the  critical 
subprocesses  were  specified  in  the  Project 
Performance  Measurement  Plan 

<^These  USL/LSL  are  compared  with  the  model  and 
the  baseline  figures  in  a  summary  table 

^The  correctness  of  these  USL/LSL  of  selected  critical 
subprocesses  are  reviewed 

^Statistical  techniques  used  include:  XmR  charts,  u- 
chart,  confidence  interval,  and  Pareto  charts  for  the 
defects  by  category 
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Quantitative  Project 
Management  Highlights  ■  5 
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The  Project  Performance  Measurement  Plan  contains 
the  description  of  the  'responsibilities',  'by/to  whom', 
'how  often',  'data  storage',  'analysis' 

^Quality  objective 

Priority 

•^Metric  -  Unit  of  Measure 
^Target 

<§>USL  -  Upper  Specification  Limit 
•#LSL  -  Lower  Specification  Limit 
•^Reason  for  Target 
•#PPM  used 
^  Remarks 
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Quantitative  Project 
Management  Highlights  ■  6 
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<$>  Operational  Definitions  found  in  the  Metrics 
Definition  Guideline  [See  OPP  Sp  1.3] 

^Name  of  metric 

EBusiness  objective 

<$>  Unit  of  measure 

^Formula 

<§> Source  of  data 

Frequency  of  collection 

^  Where  the  data  is  stored 

^ Owner  of  the  data 

<^Tool  used  to  collect  basic  measures  (automatic  collection 
for  basic  measures) 

•^Collecting  requirements  to  ensure  data  validation 
^Recommended  statistical  technique 
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Quantitative  Project 
Management  Highlights  ■  7 


Training  Consulting  &  Solutions 


^Data  was  collected  at  Project  Closure  and 
presented  to  EPG 

<$>PPBs  were  updated  by  EPG  Lead  as  appropriate 

^Metrics  Definition  Guideline  was  updated  as 
appropriate 
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^  Look  before  you  jump  -  a  thorough  initial  readiness 
check  before  starting  the  high  maturity  journey 

^  Upfront  high  maturity  training  to  the  process 

champions  so  that  they,  who  understand  the  strengths 
and  weaknesses  of  the  current  processes,  can  drive  the 
SPI 


Summary  and 
Learned 


£ 
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^  Periodic  assessment  -  a  knowledgeable  Appraisal 
Team,  mostly  from  the  organization,  can  also  direct  the 
organization  towards  the  right  direction 

^  ^alk,  talk,  talk  -  the  measurement  team  and  the 
external  consultant(s)  should  communicate  with  the 
project  managers  periodically  to  validate  the 
measurement  findings;  the  Project  managers  are 
always  the  people  best  equipped  to  ‘interpret’  the 
analysis  findings 
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Edmond  Sung 


President  &  Principal 
Consultant  of  Processis  Ltd. 

Edmond  has  more  than  25  years 
of  experience  in  the  information 
technology  and  related  service 
industry 

Edmond’s  current  focus  is  to 
assist  companies  to  improve  their 
processes,  products  and  service 
quality  using  an  effective 
combination  of  CMMI  and  Six 
Sigma. 

Edmond  has  more  than  eight 
years  of  experience  in  CMMI- 
based  training  /  consulting/  and 
appraisal  services. 
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Tim  Kasse 


CEO  and  Principal 
Consultant  of  Kasse 
Initiatives  LLC 

^  Visiting  Scientist  -  Software 
Engineering  Institute 
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Visiting  Fellow  -  Institute  for 
Systems  Science  /  National 
University  of  Singapore 

Author  of  Action  Focused 
Assessment  for  Software 
Process  Improvement 

Author  of  Practical  Insight 
Into  CMMI 


©  2011  Processis  LTD  &  Kasse  Initiatives,  LLC  Version  2.3 


BAI  GUO  ML  4  Journey  -  56 


Practical  Insight  Into 
CMMI  -  2nd  Edition 
(Sept  2008) 
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NORTHROP  GRUMMAN 


CMMI,  ISO  and  AS9100: 

An  Efficient  and  Effective  Approach 


CMMI  Technology  Users  Conference 

Denver,  CO 


LaKeisha  M.  Souter 
Al  Chatmon 

Certified  SCAMPI  Lead  Appraisers 


November  17,  201 1 


Agenda 


NORTHROP  GRUMMAN 


•  Standards:  A  Necessity  for  Doing  Business 

•  Standards  Across  Our  Organization 

•  An  Integrated  Approach 

•  Standards  Comparisons 

•  Steps  to  Developing  and  Integrated  Approach 


Compliance  Standards: 

A  Necessity  for  Doing  Business 
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Compliance  Standards: 
Driven  by  the  Business 


NORTHROP  GRUMMAN 


For  Our  \ 

Customers  X 

\  Innovative,  high-performance,  affordable  products  and  services, 

W  delivered  on  time  and  with  the  promised  performance,  quality  and 
reliability,  that  ensure  our  customers’  success  in  their  operations 

For  Our  \  \  Business  performance  that  is  predictable  and  reliable,  delivering 

Shareholders  X  X  sustained  returns  on  shareholders’  investments 

For  the  \ 

Corporation  and  y 
the  Sector  / 

\  Technical  and  business  processes  that  are  faster,  more  profitable,  and 
m  able  to  deliver  products  and  service  with  more  performance,  better 

^  quality,  and  lower  cost  than  our  competitors 

For  Our  \ 

Businesses  &  y 

Programs  X 

\  Deliverable  products  &  services  and  internal  technical  &  business 

y  processes  that  ensure  our  ability  to  meet  or  exceed  the  contract 
commitments  we  have  made. 

For  Our  \ 

Employees  X 

\  A  work  environment  that  makes  it  easy  for  employees  to  apply  their 
y  natural  talents  with  passion  and  excellence  and  to  gain  new  skills  and 

7  capabilities  that  will  open  up  future  opportunities  for  success 
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Compliance  is  the  cost  of  doing  GOOD  business 


The  CMMI®  (Capability  Maturity  Model 
Integration)  model 


NORTHROP  GRUMMAN 


It  is  a  model  with  22  inter-related 
process  areas  grouped  by  category: 
Engineering,  Support,  Project 
Management,  Process  Management 

It  is  used  to  measure  project 
management  and  development 
activities  across  project  lifecycles. 

The  CMMI  is  a  process  model  that: 

-  Is  a  collection  of  industry  best  practices 

-  Contains  a  framework  for  organizing 
and  prioritizing  process  improvement 
activities 

-  Is  used  to  emphasize  the  alignment  of 
process  improvement  objectives  and 
organizational  business  objectives 
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ISO:  The  Quality  Management  System 


NORTHROP  GRUMMAN 


Key 

- Value-adding  aelivilte& 

- ■  —  lnfarmalk>n  flow 

Figure  1  —  Model  of  a  process-based  quality  management  system 

amaSa'd Crasnbsfllor fa-£&-ica-d isBop  ®  ISO  2008  —  All  riphtS  reserved 


•  Section  4:  Quality  Management 
System 

•  Section  5:  Management 
Responsibility 

•  Section  6:  Resource  Management 

•  Section  7:  Product  Realization 

•  Section  8:Measurement,  Analysis 
and  Improvement 


We  use  ISO  to  meet  the  needs  of  customers  and  other  stakeholders. 
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AS9100 


NORTHROP  GRUMMAN 


•  AS  9100  Quality  Systems  -  Aerospace  -  Model  for  Quality  Assurance 
in  Design,  Development,  Production,  Installation  and  Servicing 

-  AS9100  is  a  widely  adopted  and  standardized  quality  management  system  for  the 
aerospace  industry 

-  The  current  version  of  AS9100  aligns  the  standard  with  ISO  9001:2008  and  has 
extra  requirements  regarding  Regulatory  Compliance  and  the  following  aerospace- 
sector  specific  requirements: 
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IS014001 


NORTHROP  GRUMMAN 


•  ISO  14001 :2004  specifies  requirements  for  an 
environmental  management  system 

•  Enable  an  organization  to  develop  and  implement  a  policy 
and  objectives  which  take  into  account  legal  requirements 
and  other  requirements  to  which  the  organization 
subscribes,  and  information  about  significant 
environmental  aspects.  It  applies  to  those  environmental 
aspects  that  the  organization  identifies  as  those  which  it 
can  control  and  those  which  it  can  influence.  It  does  not 
itself  state  specific  environmental  performance  criteria. 


ISO  TickIT 


NORTHROP  GRUMMAN 


•  ISO  TickIT  is  a  quality-management  certification  program 
for  software  development 

•  Major  objective  was  to  provide  industry  with  a  practical 
framework  for  the  management  of  software  development 
quality  by  developing  more  effective  quality  management 
system  certification  procedures.  These  involved: 

-  publishing  guidance  material  to  assist  software  organizations  interpret  the 
requirements  of  ISO  9001 

-  training,  selecting  and  registering  auditors  with  IT  experience  and 
competence,  and 

-  introducing  rules  for  the  accreditation  of  certification  bodies  practicing  in 
the  software  sector 
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One  Approach: 

Linear,  One-to-One  Compliance 


NORTHROP  GRUMMAN 


Develop  an  organizational  process  for  each  major  standard 


AS  9100  Rev.  C 

i  AS  9100  Oraanizational  Process 

CMMI  1.3 

i  CMMI  Organizational  Process 

ISO  9001:2008 

i  ISO  9001  Organizational  Process 

ISO  TickIT 

i  =^>  ISO  TickIT  Organizational  Process 

ISO  14001 

Sarbanes-Oxley  Act  of  2002 


What  do  you  do  when  there  are  multiple  compliance  requirements? 
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Our  Approach:  Integrated  Enterprise  Process 


NORTHROP  GRUMMAN 
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Improving  Program  Execution 


A  Fundamental  Competitive  Advantage 


12 


•  Complies  with  key 
standards 

•  Encourages  integration  of 
all  disciplines 

•  Eliminates  duplications 

•  Implements  “good” 
approaches  to  resolving 
“conflicts”  between 
standards 


CMMI’s  OSSP  vs.  ISO’s  QMS  (1  of  2): 
How  did  we  balance  the  two? 


NORTHROP  GRUMMAN 


CMMI  OSSP -a 
collection  of  definitions 
of  the  processes  that 
guide  activities  in  an 
organization. 


QMS  -  organization’s 
processes  for 
management  activities, 
provision  of  resources, 
product  realization, 
measurement,  analysis 
and  improvement. 
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CMMI’s  OSSP  vs.  ISO’s  QMS  (2  of  2): 
Compliance  Matrixes 


NORTHROP  GRUMMAN 


ISO  Compliance 
Matrix 


ISO  Compliance  Matrix: 
•Institutes  our  QMS 
systems 

•Maps  ISO  with  our 
processes  and 
procedures 


14 


CMMI  Evaluations  vs.  ISO  Internal  Audits 
How  did  we  balance  the  two? 


NORTHROP  GRUMMAN 


CMMI’s  Objective  Evaluation 
(PPQA)  involve: 

•Objectively  evaluating  performed 
processes  and  work  products  against 
applicable  process  descriptions,  standards, 
and  procedures 
•Identifying  and  documenting 
noncompliance  issues 
•  Providing  feedback  to  project  staff  and 
managers  on  the  results  of  quality 
assurance  activities 

•Ensuring  that  noncompliance  issues  are 
addressed 


ISO’s  Internal  Audits  are 
conducted  at  planned  intervals  to 
determine  whether  the  quality 
management  system 

•Conforms  to  the  planned  arrangements 
(product  realization  plan),  to  the 
requirements  of  the  ISO  standard,  and  to 
the  quality  management  system 
requirements  established  by  the 
organization,  and 
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CMMI  Evaluations  vs.  ISO  Internal  Audits: 
Internal  Audit  Effectiveness 


Performs  approximately 
400  internal  audits  a  year 
across  the  Baltimore 
campus  covering 
programs,  functional 
organizations,  engineering 
disciplines  and 
laboratories. 


Ensures  compliance  to 
IEP. 


Satisfies  CMMI  Process 
and  Product  Quality 
Assurance  practices  and 
ISO  internal  auditing 
requirements. 

Sunnyvale^ 

Satisfies  GP  2.9  across  th$&MSD 
CMMI  Process  Areas. 


IAE  Reporting  Sites 


Rolling  Meadows,  IL 

Salt  Lake  City,  UT  Boulder,  CO  Springs  c6*spsd 


Norwalk,  CT 
ISRSD 


Woodland  Hills,  CA 

NSD  1SRS° 


altimore,  MD 
ykesville  (PCS) 
N&MSD 

Sykesville  (FSSO) 
PAP 

Troy  Hill  (SPS) 
L&SPSD 
oy  Hill  (PAP) 
PAP 

Annapolis,  MD 
N&MSD 

Charlottesville,  VA 

N&MSD 

Apopka,  FL 
TSD 

Melbourne,  FL 
ISRSD 
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CMMI  Evaluations  vs.  ISO  Internal  Audits: 
IAE  Timeline  -  Implementation  &  Baseline 


NORTHROP  GRUMMAN 


Reviewed  campus 
internal  audit  schedules 
for  incorporation  of 
recommendations; 

Provided  feedback 
(including  requests  for 
objective  evidence) 


ES  launched  initiative  to 
evaluate  and  restructure 
its  Internal  Quality  Audit 
function  to  focus  more 
on  risk  areas  as  opposed 
to  only  emphasizing 
ISO/AS/TickIT 
certification 

2008 


Criteria  established  for 
measuring  incorporation 
of  each  recommendation 


2009 


Requested  each  division 
to  begin  to  incorporate 
enhancements  into  audit 
programs 

Dashboard  metrics 
established  for  reporting 
status 


continued... 


Capture  baseline  scoring 

Determine  program 
strengths  and 
weaknesses 


Cross-campus  Kaizen 
events  held  to  identify 
weaknesses  in  ES 
Internal  Audit  Program; 

8  recommended 
enhancements  identified 
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Incorporation  of  recommended  enhancements  =  improvement 


CMMI  Evaluations  vs.  ISO  Internal  Audits: 

IAE  Timeline  -  Implementation  &  Baseline  (cont.) 


NORTHROP  GRUMMAN 


Revised  program  for 
more  emphasis  on 
execution  and 
improvement 

Incorporated  new 
divisions  structure 

Revised  scoring  process 

2010  I 


Begin  quarterly  Sector  AE  metric  added 
visibility  and  reporting  to  Sector 

using  revised  scoring  Operating  Factors 

methodology 

Report  measured 

end  of  data  to  higher 

level  management 

end  of 

2010 


end  of  end  of  end  of 

2011,  2012,  2013, 


Measure  Effectiveness 


Request  2010  schedules 

end  of  April 


Collect, 
analyze,  and 
interpret  KPI 
data 


Generate  KPIs 
and  objectives 
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Formal  AE  Implementation  &  Measurement 


CMMI  Higher  Level  Management  Reviews  vs. 
ISO  Top  Management  Review  (1  of  2) 


NORTHROP  GRUMMAN 


ISO  Top  Management  Review: 

Top  management  shall  review  the 
organization's  quality  management 
system,  at  planned  intervals,  to  ensure 
its  continuing  suitability,  adequacy  and 
effectiveness.  This  review  shall  include 
assessing  opportunities  for 
improvement  and  the  need  for  changes 
to  the  quality  management  system, 
including  the  quality  policy  and  quality 
objectives. 


CMMI’s  Higher  Level  Management 

Reviews  : 

provide  higher  level  management  with 
appropriate  visibility  into  the  process. 
Different  managers  have  different 
needs  for  information  about  the 
process.  These  reviews  help  ensure 
that  informed  decisions  on  the  planning 
and  performing  of  the  process  can  be 
made.  Therefore,  these  reviews  are 
expected  to  be  both  periodic  and  event 
driven. 
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CMMI’s  Higher  Level  Management  Reviews  vs. 
ISO’s  Top  Management  Review  (2  of  2) 


NORTHROP  GRUMMAN 


Business  & 

Organizational 

Objectives 


Mis: 

Assur 

Vii 

Pres 

;ion 

•a  nee 

ce 

dent 

Mission 

Assurance 

Directors 

Program  & 
Department 
Measurements 
including  Process 
Effectiveness 
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Steps  to  Developing  a  Multi-Standard  Compliant  NORt„Rop crummy 
Organizational  Process 


1 .  Assess  the  various  process  architectures/frameworks  and  decide 
which  is  best  for  your  organization. 

2.  Identify  the  major  process  elements  that  comprise  your 
organizational  process. 

-  Consider  the  Process  Areas  of  CMMI,  but  don’t  overlook  other  important 
elements  that  may  be  significant  to  your  business. 

-  Consider  the  needs  of  the  various  disciplines  required  for  your  business. 

-  Develop  a  process  “model”  identifying  the  order  of  execution  of  the 
process  elements.  Note:  there  maybe  more  than  one  order  required.  Each 
discipline  may  have  a  more  detailed  process  “model”.. 

3.  For  each  process  element,  identify  the  most  stringent  standard. 
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Steps  to  Developing  a  Multi-Standard  Compliant 
Organizational  Process 


NORTHROP  GRUMMAN 


4.  Develop  the  process  element  description  to  meet  the  requirements  of 

the  most  stringent  standard. 

-  Attempt  to  retain,  or  slightly  modify  when  necessary,  the  current  practices  that 
are  working  for  the  organization. 

-  Develop  new  process  only  when  absolutely  necessary  to  comply.  Note:  we 
don’t  recommend  doing  other  process  improvements  at  the  same  time . 

-  Build  a  matrix  or  equivalent  identifying  how  compliance  is  achieved. 

-  Include  the  appropriate  process  user  representatives  in  the  review  activities. 

-  Resolve  discovered  issues. 

5.  Validate  that  compliance  is  achieved. 

-  Include  an  expert  of  the  standard  and  quality. 

-  Resolve  discovered  issues.  Resolution  may  require  further  review  by  the 
process  user  representation. 

6.  Check  to  see  if  the  other  applicable  standards  are  also  achieved. 

7.  Integrate  the  process  elements.  Consistent  with  the  process  “model” 
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Steps  to  Developing  a  Multi-Standard  Compliant 
Organizational  Process 


NORTHROP  GRUMMAN 


8.  Check  to  see  if  the  other  applicable  standards  are  also  achieved. 

-  If  not,  amend  the  process  element  description  appropriately  ensuring  that 
compliance  to  the  detailed  standard  is  not  lost. 

-  Build  matrices  or  equivalents  identifying  how  compliance  is  achieved. 
Note:  each  standard  will  have  its  own  matrix  or  equivalent. 

-  Include  the  process  user,  the  standard  expert,  and  quality  in  the  review 
activities. 

9.  Integrate  the  process  elements.  Consistent  with  the  process  “model”: 

-  Ensure  that  the  inputs  to  each  process  element  are  in  fact  created  by 
another  process  or  available  from  a  library,  reference,  or  storage. 

-  Ensure  that  the  outputs  from  each  process  element  supports  follow-on 
activities  or  challenge  its  need. 

-  If  the  integration  drives  changes  to  the  process  element,  ensure  the 
applicable  compliances  are  sustained  and  update  the  compliance 
matrices  as  required. 

•  Standard  experts,  process  users  and  quality  should  participate  in  the 
review  of  changes  as  required. 
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Summary 


NORTHROP  GRUMMAN 


•  Rank  compliance  standard  in  order  of  importance  to  your  business 

•  Leverage  similarities  between  the  standards 

•  Think  Organizationally 
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NORTHROP  GRUMMAN 


<OST 

CMMI  LEVEL  5  |  ISO  9001:2008 
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Agenda 


<  Purpose 

<S  Background 

<  Let’s  talk  about  Change 

<S  Our  Change  Management  Plan 
<S  Change  Management  Team 
<S  Conclusions 
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Purpose 


What  is  Change  and  how  can  a  small  business  achieve  Process 
Improvement  Change? 

OST’s  case  study  on  achieving  change 

►  Our  Change  Management  Plan  &  transition  to  a  high  maturity  organization 

►  Our  Change  Management  team 

•  How  we  defined  our  80/20 

•  Our  early  adopters 

•  What  to  do  with  detractors? 
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Background  -  Who  we  are 


OST,  Inc 

«  Washington  DC-based,  founded  in  1999 
Core  competencies 

►  Integrated  IT  solutions 
Managed  Services 
Management  consulting 
Research,  development  &  engineering 

CMMI  L5  (CMMI-DEV  vl.2) 

<  ISO  9001:2008  certified 

<  ANSI  748  compliant 


4 


<OST 

CMMI  UVB  5  |  ISO  9001  2008 


Managed 

Services 


<OST 


CMMI  LEVEL  5 

ISO  9001:2008 


Y  Research,  \ 
Development, 
l  &  Engineering  j 


1  Integrated  l 

Process 

f  Management  1 

l  IT  Solutions  J 

Improvement 

1  Consulting  J 

OUR  CERTIFICATIONS 

j  I  CMMI 

EVM 


ITIL 


ANSI/EIA  748 


»0«lDWIDE 


Offices  Located:  DC,  Virginia,  Maryland,  Pennsylvania  and  Washington. 


Background  -  Our  Process  Improvement  Timeline  <OST 

CMMI  l,  EVE  l  5  |  ISO  9001  2008 


<OST 


PROCESS  IMPROVEMENT  PROGRESS 


-  CMMI  Level  3  GAP  Analysis, 
Transition  Planning  &  SPIP 

-  CMMI  Level  3  Training 

-  PAL  reorganized 


Successful  ISO  9001:2008 
Audit  (Sep  2010) 


Successful  ISO  9001:2008 
Audit  (Sep  2009) 


Successful  CMMI  Level  5 
Appraisal  (Jan  2011) 


0ST  Externally  Assessed 
at  SW-CMM  Level  3  (Aug  2003) 
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Background-  Our  Process  Improvement  Structure 


CMMl  UVB  5  I  ISO  9001  2006 


Business  Unit 
Leads  (BUL) 


ftttttftttttt 

Process  Users/Practitioners 


w 


Change  Agents 


SEPG 

nr 


Process  Action  Teams  (PATs) 
Business  Performance  Group  (BPG) 
Quality  Assurance  Group  (QA) 
Engineering  Review  Board  (ERB) 
Other  focus  groups 


Projects 


6 


What  is  change? 
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Change  @  OST  -  A  case  study 


◄  Wanted  to  apply  lean  concepts 
Use  smartly  limited  resources 

Use  existing  framework 


<  Wanted  to  get  away  from  the  gut-feeling  improvements 


To  proven-improvements 

◄  Competitive  advantage 
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Top  down-  Senior  Management’s  Support 


<OST 


:mmi  I  E VEl  5  I  ISO 


◄  They  saw  the  value  from  the  beginning 
•<*  Communicated  the  main  message 

►  Towards  the  company’s  vision 

►  Benefits  of  the  change  at  different  levels 


<  Maintained  the  enthusiasm  and  motivation 
•<*  Head  of  communication  channel 
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Change  Management  Plan 


<OST 


:MMI  I  fVFl  5  ISO  9001  2( 


r 

Collect 

iproveme 

nformatioi 

1 

r 

r 

Review 

[: 

Status  with 
Higher 
.evel  Mgt. 

J 

L 

A 

Objectively 

Evaluate 

Adherence 

L 

A 

Identify  and 
Manage 
Stakeholders 
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The  80/20-  Our  people 


<OST 

CMMl  I, EVE l  S  I  ISO  9001  2006 
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How  to  define  your  80/20  -  Bringing  change 


<  Crossing  the  chasm  - 


r 

The  Early  Adopters 

20% 


P.l.  Visionaries 
Ethusiasts 


V  ^ 


The  Regular  Practitioners  The  Detractors 

- 80% - 


The 

Chasm 


Early  Majority  Late  Adopters 


Skeptics 


Source:  Crossing  the  Chasm  Geoffrey  A  Moore.  Collins  Buxine 
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◄  Creation  of  Business  Performance  Group  (BPG) 

►  Group  to  take  process  improvement  to  the  next  level 

•  “Early  adopters”  from  several  projects  and  backgrounds 

►  SEPG  leaders  selected  the  team 

•  Interaction  from  governance  activities,  audits,  focus  group  participations 

◄  Characteristics 

•  Humble  -  Attitude  counts 

•  Hungry 


-  Challenge  you 

-  Ownership 

-  Leadership 


•  Trustworthy 

•  Availability  and  Commitment 

◄  Got  buy-in  from  respective  PMs  and  Business  Unit  Leads 

►  Before  talking  to  potential  members 
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Change  Management  Team-  From  Group  to  Team 


Invested  time  for  the  members  to  know  each  other 

►  Myers-  Briggs  -  Learn  how  to  work  among  team  members 

►  Meet  in  person  as  often  as  possible 

►  Build  team  values 

<  Let  them  know  about  the  distinction 

►  What  they  bring  to  the  table 

►  Let  their  peers  know 

►  Sr.  Mgt  reinforced'  the  message  and  kept  them  motivated 


<  Gave  them  authority  for  their  area  of  work 

<  Let  them  take  ownership  in  tasks  in  their  area  of  interest 

►  Sometimes  challenged  them  with  other  tasks 
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Change  Management  Team  -  Responsibilities 


Vision 


Transition 


Alignment 


Empowerment 


Resources 


Dialogue 


Congruence 


Anticipation 


Change  Management  Team-  What  they  accomplished 


Planned  and  implemented  change  management  action  plans 

►  Provided  feedback  from  the  practitioner’s  perspective 

►  Ensured  ease  of  use  and  usefulness*  of  new  processes 

<  Led  by  example 

►  Implemented  on  their  projects 

►  Introduced  change  to  their  peers 

►  Established  the  buddy  program  to  ensure  understanding 

•<S  Became  the  main  channel  of  communication 

►  Gather  first  source  feedback  > 

►  Gained  the  trust  of  other  peers  > 

►  Are  still  supporting  implementation  of  L5 
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*  Medha  Umarji,  Gauging  acceptance  of  software  metrics:  Comparing  perspectives  of  managers  and  developers 
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Detractors?-  What  to  do 


Identified  skeptics 

Through  the  feedback  provided 

►  Their  attitude  toward  process  changes 

<  Analyze  their  feedback 

►  Found  the  root  cause 

•  Commonalities-  role,  activities,  concerns 

►  Incorporated  feedback  in  the  process 

►  Establish  action  plans  to  get  buy-in 

<  PMs  shared  concerns 


►  Project  Managers  that  had  concerns  about  sustainability 

►  Impact  on  the  project  constructs 

►  Created  a  group  with  early  adopter  PMs  and  BULs 

•  Helped  us  getting  understanding  of  where  others  were  coming  from 
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Change  Management-  Did  it  work? 


<OST 


i  I  'SO 


Practitioners  start  driving  the  change 

►  Make  requests 

►  Provide  feedback 
Provide  new  ideas 

•<S  Their  language  changed 

►  Practice  interviews-  L5  institutionalized 

►  Practitioners  started  inquiring  to  implement  in  their  projects 

•<S  Our  business  outlook  changed 

►  Strategic  and  goal  oriented 

►  Replaced  gut  feeling  with  quantitative  decision  making 
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What  we  learned 


<OST 


I  ISO  ^ 


◄  If  you  are  having  a  bumpy  road... 

A  lot  of  noise  is  coming  back 

Don't  worry,  it  shows  they  are  listening 


Don't  assume  all  resistance  comes  from  being  a  detractor... 
Most  times  is  miscommunication 


< 

<s 

< 


Don't  wait  for  a  fully  cooked  recipe  to  share... 
Share  a  little  bit  on  the  way  to  get  feedback  and 

Keep  your  cool  and  remember  your  core  values. 
People  drive  the  change 

Don’t  forget  about  the  middle 


What  we  learned 


<OST 


20 


Conclusion-  The  Pursuit  of  True  Change 


People,  process,  and  technology  impact 
quality 


<  Process  is  a  sequence  of  steps 
performed  for  a  given  purpose  (ieee) 


Process  changes  have  the  most  impact 
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Conclusion-  A  Suggested  Roadmap 


k  A 


Define  the  Battle 


Assemble  and 
Invasion  Force 


Launch  the 
Invasion 


Why  are  you  pursuing  the  change? 
Does  it  align  to  your  vision? 

Define  benefits  at  all  levels 

Get  Senior  Management’s  commitment 

Create  a  Plan-  Use  your  GPs 


Create  the  “whole  product’’-  processes,  solutions,  support  structure,  etc 
Ensure  its  ease  of  use  and  usefulness-  Otherwise,  who  wants  it? 
Establish  support  mechanisms-  trainings,  buddy  programs,  etc 
Ensure  the  outputs  provide  the  expected  benefit 


Find  your  early  adopters-  They  will  become  your  best  sellers  and  provide  feedback 
Create  a  Change  Management  Team  -  for  practitioners  and  PM  level 
Find  your  skeptics-  Analyze  why  and  how  to  bring  them  over 


f 

A 

•  Define  and  manage  channels  of  communication 

•  Use  your  change  management  team  as  your  “sales  force” 

•  Beat  the  drums-  Sr.  Mgt,  SEPG,  Practitioners 

J 
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Summary 


✓ 


What  is  Change  and  how  can  a  small  business  achieve  PI  Change? 

OST’s  case  study  on  achieving  change 

Our  Change  Management  Plan  &  transition  to  a  high  maturity  organization 
■s  Our  Change  Management  team 
s  How  we  defined  our  80/20 
s  Our  early  adopters 
s  What  to  do  with  detractors? 
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Questions  &  Answers 


Any  Questions? 
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<OST 

CMMI  I.EV61  5  |  ISO  9001  2008 


Appendix-  Reference  Material 


<  Crossing  the  Chasm :  Geoffrey  A.  Moore,  Collins  Business  Books  essentials, revised 
Edition,  2002.  ISBN  13:  9780060517120  ISBN  10:  0060617123 

<  Gauging  acceptance  of  software  metrics:  Comparing  perspectives  of  managers  and 
developers,  Medha  Umarji,. 

http://www.spamcast.libsvn.com/index.php7post  cateqory=Measurement 
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MDD  VI. 3  Glossary  Definitions 

■  Basic  unit  -  A  managed  set  of  interrelated  resources  which  delivers  one  or 
more  products  or  services  to  a  customer  or  end  user  and  typically  operates 
according  to  a  plan  (e.g.,  projects,  work  groups). 

■  Sampling  factor  -  organizational  or  work  context  that  reflects  meaningful 
differences  in  the  way  work  is  performed  across  different  basic  units  within 
the  organizational  unit  (e.g.,  size,  location,  customer). 

■  Subgroup  -  Cluster  of  basic  units  that  share  common  sampling  factor 
alternatives  and  exhibit  similar  process  implementations. 

■  Support  function  -  An  organizational  group  that  provides  products  and/or 
services  for  a  bounded  set  of  activities  needed  by  other  portions  of  the 
organization  (e.g.,  Configuration  Management  group,  Quality  Assurance 
group). 

■  Organizational  scope  -  The  collection  of  basic  units  and  support  functions 
that  provides  instantiations  of  practices  used  within,  and  representative  of, 
an  organizational  unit. 
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Sampling  Factors  in  Action 
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Determining  Subgroups  and  Samples  Example:  BINDY  Co. 

1 .  Identify  Sampling  Factors 

■  Location:  Indianapolis,  Boston 

■  Type  of  Work:  new,  maintenance 

■  Customer:  DoD,  commercial 

2.  Combine  sample  factors,  sort  basic  units  (BUs),  determine  min.  sample 

■  Minimum  #  of  BUs  per  subgroup  =  (#  BUs  in  subgroup  x  #  subgroups)  /  total  #  BUs 


BINDY  Co. 

Location 

Type  of 
Work 

Customer 

#  of  BUs  in 
subgroup 

#  subgroups  X  #  BUs 
in  subgroup 

...divided  by  total  # 
BUs 

Min.  Number 
Sampled 

Boston 

new 

comm 

0 

0 

0.00 

0 

Subgroup  1 

Boston 

new 

DoD 

4 

20 

0.13 

1 

Subgroup  2 

Boston 

maint 

comm 

49 

245 

1.64 

2 

Boston 

maint 

DoD 

0 

0 

0.00 

0 

Subgroup  3 

Indy 

new 

comm 

5 

25 

0.17 

1 

Subgroup  4 

Indy 

new 

DoD 

16 

80 

0.54 

1 

Indy 

maint 

comm 

0 

0 

0.00 

0 

Subgroup  5 

Indy 

maint 

DoD 

75 

375 

2.52 

3 

Totals 

5 

149 

8 

#sampling  factors  =>  #  subgroups  =>  #  BUs  sampled 
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Outline 


Raytheon 


■  Introduction 

■  CMMI  VI. 3  Appraisals 

■  Sampling  Factors 

■  Subgroups 

■  Basic  Units 

■  Support  Functions 

■  Organizational  Unit  Size 

■  Data  Relationships 

■  SCAMPI  Infrastructure  Implications 

■  Comparing  MDD  VI. 2  and  MDD  VI. 3  CMMI  VI. 3  Appraisals 

■  Coda 
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Introduction  -  Purpose 

■  Understand  adoption  and  usage  of  MDD  VI. 3 

■  Provide  guidance  to  future  MDD  VI  .3  users 

■  Ensure  MDD  VI. 3  is  working  and  being  used  as  intended 

■  Identify  possible  SCAMPI  infrastructure  implications  based 
on  usage  within  the  community 

-  MDD,  SAS,  PARS,  ADS,  Training  Material 
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Introduction  -  Briefing  Contents 

This  briefing  includes: 

-  MDD  VI. 3  Adoption 

-  Sampling  Factor  Analysis 

-  Subgroup  Analysis 

-  Basic  Unit  Analysis 

-  Support  Function  Analysis 

-  Organizational  Unit  Analysis 

-  Data  Relationships 

-  SCAMPI  Infrastructure  Implications 

-  Comparing  MDD  VI  .2  and  MDD  VI  .3  CMMI  VI  .3  Appraisals 


■  Notes: 

-  All  data  comes  from  the  SEI  Published  Appraisal  Results  Site  (PARS) 

-  Data  includes  SCAMPI  A  appraisals  posted  to  PARS  by  November  1, 2011 
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MDD  VI  .3  Adoption  Data 


since  MDD  V1.3  release 

87 

70% 

37 

30% 

16 

76% 

5 

24% 

0 

0 

103 

71% 

42 

29% 

Nov  2010 

Dec  2010 

Jan  2011 

Feb  2011 

Mar  2011 

Apr  2011 

May  2011 

Jun  2011 

Jul  2011 

Aug  2011 

Sep  2011 

Oct  2011 

total 

CMMI-DEV  V1.3  MDD  V1.2 

0 

1 

3 

2 

7 

12 

15 

15 

17 

12 

14 

2 

100 

73% 

CMMI-DEV  V1.3  MDD  V1.3 

0 

0 

0 

0 

0 

1 

2 

4 

9 

5 

13 

3 

37 

27% 

CMMI-SVC  V1.3  MDD  V1.2 

1 

1 

0 

0 

1 

2 

3 

4 

1 

3 

3 

0 

19 

79% 

CMMI-SVC  V1.3  MDD  V1.3 

0 

0 

0 

0 

0 

1 

0 

0 

1 

1 

2 

0 

5 

21% 

CMMI-ACQV1.3  MDD  V1.2 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

100% 

CMMI-ACQ  V1.3  MDD  V1.3 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0% 

Total  CMMI  V1.3  MDD  V1.2 

1 

2 

3 

2 

9 

14 

18 

19 

18 

15 

17 

2 

120 

74% 

Total  CMMI  V1.3  MDD  V1.3 

0 

0 

0 

0 

0 

2 

2 

4 

10 

6 

15 

3 

42 

26% 

I  Total  CM  Ml  VI. 3  MDD  VI. 3 
I  Total  CM  Ml  VI. 3  MDD  VI. 2 


Nov  Dec  Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct 

2010  2010  2011  2011  2011  2011  2011  2011  2011  2011  2011  2011 


Notes: 

1.  47  MDD  VI  .3  appraisals 
reported  in  PARS,  42 
with  CMMI  VI  .3  models 
(see  graph/data). 

2.  5  CMMI  VI. 2  MDD  VI. 3 
appraisals  were  reported 
(not  shown  on  this  chart), 
2  in  1  multi-model 
appraisal. 

3.  No  P-CMM  MDD  VI  .3 
appraisals  have  been 
conducted. 

4.  2  CMMI  VI. 3  MDD  VI. 3 
multi-model  appraisals 
were  conducted. 
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Sampling  Factors 


0  12  3 

#  Sampling  Factors 


Sampling  Factors 


a 

'(Z 

i— 

Q. 

Q. 

< 


20 

18 

16 

14 

12 

10 

8 

6 

4 

2 

0 
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#  Sampling  Factor  Values 

A  ve.  =  2.2 

□  2  □  3  D4  □  5  □  6  Ml  B8 

Med.  =  2 

2%  -^C)%  2% 

/Cx  tl^l 

/  13%  \yj  \ 

V  J 

Notes: 

1 .  No  appraisals  have  >  3  sampling  factors. 

2.  8  of  the  10  “zero-sampling  factor”  appraisals 
included  100%  of  OU  basic  units. 

3.  [Type  of  Work,  Customer,  Org  Structure, 

Size]  constitute  77%  of  sampling  factor 
usage  (not  counting  N/As). 

4.  “Location”  (required  to  be  considered)  is  not 
typically  a  sampling  factor. 

5.  Sometimes  “sampling  factors”  appear  to  be 
“forced”  (e.g.,  “Customer”  identified  as  a 
sampling  factor,  but  description  says 
“Though  the  OU  has  different  customers,  the 
work  is  performed  the  same  way  for  all 
customers.” )  SEI  Appraisal  System  problem. 

- — -  1 1/21/201 1  I  8 - 


Subgroups 
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Subgroups  in  Appraisals  m  V3? 


# Subgroups 


Notes: 

1 .  If  average  #sampling  factors  =  2  and  average  #sampling  factor  values  =  2,  an  average  of  3-4 
subgroups  seems  reasonable. 

2.  One  appraisal  had  2  sampling  factors,  each  with  2  sampling  factor  values,  but  only  1  subgroup. 

Questionable  data:  “forced”  sampling  factors. 

3.  In  5  appraisals,  the  #basic  units  in  the  organizational  scope  is  less  than  the  #subgroups.  Questionable 
data:  #basic  units  should  be  >  #subgroups  (MDD  Data  Coverage  Rule  1  for  Basic  Units). 

•  In  4  of  the  5  appraisals,  support  functions  (QA,  HR,  Training,  EPG)  were  being  called  subgroups,  and 
sometimes  also  called  sampling  factors  and/or  sampling  factor  values.  Questionable  data:  MDD 
definition  of  subgroup  is  “Cluster  of  basic  units  that  share  common  sampling  factor  alternatives. 

•  In  1  “zero-sampling  factor  appraisal”,  100%  of  the  OU  was  contained  in  the  organizational  scope,  and 

was  segmented  into  “subgroups”  that  don’t  align  with  MDD  usage  of  the  term  subgroup.  Questionable 
data:  misuse  of  the  term  subgroup.  11/21/2011  |  9 


Basic  Units 
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Basic  Units  in  Appraisals 


Ave  =  4.8 
Med.  =  4 


ra 

i. 

a 

a 

< 


1 1 

,  1 1 

.1 

i^i  i  i  ^  i  i  i  i  i  i  i 

Basic  Units 


Notes: 

1 .  If  the  average  #subgroups  =  3,  then  the  average  #basic  units  =  4-5  seems  reasonable. 

2.  The  “18  basic  unit”  example  was  a  large  organization  (2000+),  with  6  subgroups,  some  of 
which  contained  as  many  as  100  basic  units). 

3.  The  “19  basic  unit”  example  was  a  small  (<  100)  organization  whose  appraisal  also 
contained  5  support  functions  and  6  subgroups.  15  basic  units  had  just  1  person  each.  4 
of  the  5  support  functions  had  just  2  people  each. 

4.  The  “20  basic  unit  example”  was  a  large  organization  (2000+)  that  chose  to  broaden  OU 
coverage  beyond  MDD  VI. 3  minimum  requirements  in  order  to  include  projects  of  strategic 
importance  and  to  include  projects  from  various  locations  even  though  location  was  not 
considered  a  sampling  factor. 

5.  47%  of  appraisals  still  have  3  or  fewer  basic  units  (an  MDD  VI. 2  concern).  v,u 


1  10 


Support  Functions 
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Support  Functions  in  Appraisals  ^ve  2  9 


0123456789  10 

Support  Functions 


Notes: 

1 .  Type  of  support  function  is  not  always  entered  into  PARS 
(sensitive).  29  appraisals  contained  some  identifying  information 
about  the  support  functions. 

2.  QA,  CM,  Training,  and  EPG  are  the  most  common  support 
functions. 

3.  MA  identified  as  a  support  function  3  times. 

4.  In  some  cases,  support  functions  were  being  called  subgroups, 
and  sometimes  also  called  sampling  factors  and/or  sampling 
factor  values.  Questionable  data. 
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Organizational  Unit  Size 


Organizational  Unit  Size  1-100-65% 

101-200-15% 
201-2000+ -20% 


1001-2000 


Notes: 

1 .  MDD  VI  .3  OU  Size  Percentages  are  comparable  to  CMMI 
Maturity  Profile  data. 

2.  OU  size  is  estimated  based  on  basic  unit  and  support  function 
“number  of  people”,  and  “%  of  people  included”  fields  in  PARS. 


CMMI  3/2011 
Maturity  Profile 

1-100 

61% 

101-200 

17% 

201-2000+ 

22% 

OUSize 

# 

Appraisals 

<25 

6 

26-50 

15 

51-75 

6 

76- 100 

3 

101  -  200 

7 

201  -  300 

2 

301  -  500 

3 

501  - 1000 

1 

1001  -  2000 

1 

2000+ 

3 

total 

47 
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Data  Relationships  (i  of 4) 

#Sampling  Factors  and  #Basic  Units  =  f(OU  Size  )?...too  early  to  tell. 
Current  data  does  not  show  a  relationship. 


OU  Size 

# 

Appraisals 

#  Sampling  Factors  in  Appraisals 

Basic  Units  in  Appraisals 

<25 

6 

0 

1 

0 

1 

0 

0 

2 

1 

2 

3 

1 

1 

26-50 

15 

0 

2 

2 

2 

1 

1 

1 

1 

3 

2 

1 

2 

2 

2 

1 

1 

4 

3 

3 

2 

2 

6 

6 

2 

3 

3 

5 

3 

4 

3 

51-75 

6 

2 

0 

3 

1 

2 

1 

3 

5 

19 

6 

4 

4 

76-100 

3 

2 

0 

2 

3 

1 

7 

101-200 

7 

3 

3 

3 

2 

3 

2 

3 

6 

6 

5 

6 

6 

2 

5 

201-300 

2 

2 

1 

4 

5 

301-500 

3 

3 

1 

0 

6 

6 

3 

501-1000 

1 

0 

5 

1001-2000 

1 

1 

10 

2000+ 

3 

2 

2 

0 

20 

18 

2 
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Data  Relationships  (2  of 4) 

#Basic  Units  =  f(#Sampling  Factors)?... too  early  to  tell. 

•  #Basic  Units  appears  to  be  rising  as  #Sampling  Factors  rises 

•  Nominally  #Subgroups  =  f(#sampling  factors,  #sampling  factor  values)  and  #Basic 
Units  =  f(#Subgroups). 
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Data  Relationships  (3  of 4) 

#Basic  Units  =  f(#Subgroups)? 

•  Nominally  #Subgroups  =  f(#sampling 
factors,  #sampling  factor  values)  and  #Basic 
Units  =  f(#Subgroups)... 

•Data  in  red  are  questionable 

•  In  5  appraisals,  the  #Basic  Units  in  the 
organizational  scope  is  smaller  than  the 
#Subgroups. 

•Coverage  Rule  1  for  Basic  Units:  For 
each  subgroup,  both  artifacts  and 
affirmations  shall  be  provided  for  at 
least  one  basic  unit  for  every  process 
area  implemented  by  basic  units  in  the 
subgroup. 

•2  appraisals  have  2  sampling  factors 
but  only  1  subgroup.  Questionable 
data  -  forced  sampling  factor. 
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Sampling  Factors 

Subgroups 

Basic 

Units 

Average 

(BUs) 

Median 

(BUs) 

0 

1 

1 

2.3 

'  2 

0 

1 

5 

0 

1 

2 

1 

1 

1 

0 

1 

2 

0 

1 

1 

5 

0 

1 

3 

0 

1 

2 

0 

1 

1 

2 

0 

2 

5 

r  4.3 

3 

1 

2 

10 

1 

2 

2 

2 

2 

3 

1 

2 

6 

1 

2 

3 

1 

2 

3 

1 

2 

5 

1 

2 

4 

i  1 

2 

3 

2 

2 

3 

1 

3.6 

'  3 

2 

3 

3 

2 

3 

4 

2 

3 

3 

2 

3 

3 

2 

3 

4 

2 

3 

3 

2 

3 

4 

3 

3 

6 

1 

4 

6 

3 

4 

5 

3 

4 

5 

2 

4 

4 

0 

5 

6 

1 

5 

6 

1 

5 

6 

3 

5 

6 

3 

5 

6 

3 

2 

6 

18 

2 

6 

7 

2 

3 

2 

12 

20 

3 

12 

19 
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Data  Relationships  (4  of  4) 

#Basic  Units  =  f(#Subgroups) 

With  questionable  data  removed,  a  reasonably  strong  relationship  emerges 


11/21/2011 


16 


Raytheon 


SCAMPI  Infrastructure  Implications  (iof2) 

Possible  MDD  changes 

■  If  “location”  continues  to  be  a  little-used  sampling  factor,  consider  removing  it  from  list  of 
mandatory  sampling  factors  to  be  considered. 

■  The  MDD  does  not  address  “zero-sampling  factor”  appraisals. 

-  Using  the  MDD  definition  of  subgroup  (“Cluster  of  basic  units  that  share  common  sampling  factor 
alternatives...”),  zero  sampling  factors  implies  #subgroups  =  0... 

-  ...but  if  #subgroups  =  0,  there  are  no  minimum  basic  unit  sampling  requirements  in  the  MDD. 

-  Most  “zero-sampling  factor”  appraisals  shows  #subgroups  =  1 ,  but  one  case  =  5  and  another  =  2. 

-  Recommendation: 

■  Amend  the  definition  of  “subgroups”  to  allow  for  cases  where  there  are  no  meaningful  differences  in  the 
entire  OU  process  implementation  (i.e.,  #sampling  factors  =  0). 

■  Note  that  if  #sampling  factors=0,  #subgroups=1  (i.e.,  the  “subgroup”  is  the  whole  OU). 

■  Clarification  appears  to  be  needed  in  the  MDD  (or  training  material  or  a  bulletin)  on  the 
usage  of  sampling  factors,  subgroups,  basic  units,  and  support  functions 

-  Support  functions  are  not  subgroups. 

-  Sampling  factors  should  not  be  “forced”  to  create  subgroups. 

-  The  #basic  units  should  be  >  the  #subgroups  (per  Coverage  Rule  1  for  Basic  Units). 

-  If  #sampling  factors  =  0,  #subgroups  =  1 . 

-  If  #sampling  factors  >  0,  #subgroups  >  2  (because  each  sampling  factor  has  at  least  2  values). 

■  Need  to  keep  watching  key  data  relationships  to  ensure  MDD  is  working  as  intended  (Sampling 

Factors,  Subgroups,  Basic  Units,  OU  Size...)  , 

°  11/21/2011  17 
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SCAMPI  Infrastructure  Implications  (2  of  2) 

SAS/PARS/ADS 

■  Use  recommended  guidance  for  “zero-sampling  factor”  appraisals. 

-  If  #sampling  factors=0,  #subgroups=1  (i.e.,  the  “subgroup”  is  the  whole  OU). 

-  SAS  Workaround 

■  Set  #subgroups  =  1 

■  Identify  1  sampling  factor  with  2  values  (all  basic  units,  no  basic  units) 

■  May  need  some  cross-checking  of  data 

-  Several  data  entries  appear  questionable: 

■  One  entry  has  2  sampling  factors,  each  with  2  sampling  factor  values,  but  only  1  subgroup. 

■  One  entry  has  6  subgroups  but  only  two  basic  units. 

-  The  #basic  units  should  be  >  the  #subgroups  (per  Coverage  Rule  1  for  Basic  Units). 

-  If  #sampling  factors  >  0,  then 

■  the  #values  for  each  sampling  factor  >  2. . . 

■  and  #subgroups  >  2... 

■  and  #basic  units  >  2. 

■  For  example,  an  OU  identifies  only  1  sampling  factor,  containing  2  values: 

-  Customer  (commercial,  military) 

-  Subgroup  1  =  commercial,  with  at  least  one  basic  unit  in  the  organizational  scope 

-  Subgroup  2  =  military,  with  at  least  one  basic  unit  in  the  organizational  scope 


The  ability  to  filter  PARS  data  by  “Appraisal  Method”  would  facilitate  analysis. 
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Comparing  MDD  VI  .2  and  MDD  VI  .3 
CMMI  VI  .3  Appraisals  (1  Of  2) 


Raytheon 


Notes: 

1 .  Total  data  include  1 20  MDD  VI  .2  appraisals  and  42  MDD  VI  .3  appraisals. 

2.  Distribution  of  Basic  Units  in  MDD  VI  .3  CMMI  VI  .3  appraisals  is  more  even  than  MDD  VI  .2,  which 
spikes  at  3  Basic  Units. 

3.  Percentage  of  CMMI  VI  .3  appraisals  with  <  3  Basic  Units  is  virtually  the  same  for  MDD  VI  .3  (45%)  and 
MDD  VI  .2  (43%). 

4.  CMMI  VI  .3  appraisals  with  <  3  Basic  Units  sampled  (52  MDD  VI  .2  data  points,  1 9  MDD  VI  .3  data 
points)  have  comparable  OU  coverage  percentage  using  MDD  VI  .2  or  MDD  VI  .3  (difference  is  not 
statistically  significant). 
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Comparing  MDD  VI  .2  and  MDD  VI  .3  Raytheon 

CMMI  VI  .3  Appraisals  (2  of  2) 


#  CMMI  V1.3 
Appraisals 

#  Basic  Units 
(Ave.) 

#  Basic  Units 
(Median) 

%  of  OU 
Projects  (Ave.) 

%  of  OU 
Projects 
(Median) 

%  of  OU  People 
(Ave.) 

%  of  OU  People 
(Median) 

MDDV1.2 

120 

5 

4 

60% 

60% 

63% 

64% 

MDDV1.3 

42 

5 

4 

65% 

67% 

67% 

69% 

Overall  Observations 


1 .  Organizational  Scope  of  CMMI  VI  .3  MDD  VI  .2  and  MDD  VI  .3  appraisals  are  very 
similar. 

•  #  Basic  Units  included  in  the  organizational  scope  are  comparable. 

•  Differences  in  %  of  OU  projects  and  people  included  in  the  appraisal  are  not 
statistically  significant. 

2.  The  use  of  sampling  factors  in  MDD  VI  .3  appraisals  provides  more  insight  into  how 
the  basic  unit  sample  set  was  determined. 

3.  The  Published  Appraisal  Results  site  provides  no  information  on  objective  evidence 
coverage. 

•  Cost  differences  between  MDD  VI  .2  and  MDD  VI  .3  appraisals  cannot  be 
determined  from  this  analysis  alone. 

•  Organizational  scope  does  not  appear  to  be  a  cost  driver.  Impact  of  MDD  VI  .3 
data  coverage  rules  would  need  to  be  examined. 
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Raytheon 


Kudos  to  the  “innovators”  and  “early  adopters”  who  have  used  MDD  VI  .3,  paving 
the  way  for  the  rest  of  the  community! 


Innovation  Adoption  Lifecycle 


Adopter 

Category 

Description 

Innovators 

Venturesome,  interested  in  new  ideas,  risk  takers 

Early  Adopters 

More  discrete  in  adoption  choices  than 
innovators.  Realize  judicious  choice  of  adoption 
will  help  them  maintain  central  communication 
position. 

Early  Majority 

Slower  in  the  adoption  process,  have  contact  with 
early  adopters,  and  seldom  hold  positions  of 
opinion  leadership. 

Late  Majority 

Will  adopt  an  innovation  after  the  average 
member  of  the  society,  skeptical  about  an 
innovation,  in  contact  with  others  in  late  majority 
and  early  majority,  very  little  opinion  leadership. 

Laggards 

Last  to  adopt  an  innovation,  have  an  aversion  to 
change-agents,  tend  to  be  focused  on  “traditions”, 
very  little  to  no  opinion  leadership. 

Source:  Wikipedia  description  of  Everett  Roger’s  Diffusion  of  Innovations. 
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Questions 
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Contact  Information 


Raytheon 


■  For  future  questions  the  presenter  contact  information  is: 


Michael  Campo 

Email:  Michael_J_Campo@raytheon.com 
Phone  number:  978.858.5939 
Raytheon  Integrated  Defense  Systems 
Tewksbury,  MA01876 
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Raytheon 


Presenter  Biography 


■  Michael  Campo  is  a  Principal  Engineering  Fellow 
at  Raytheon  Company,  with  33  years  experience 
that  includes  roles  as  a  software  developer, 
software/system  integrator,  manager,  software 
project  manager,  and  process  group  leader.  As 
process  group  leader  for  Raytheon  Integrated 
Defense  Systems,  Mike  developed  and  deployed 
processes  that  led  to  achievement  of  CMMI 
Maturity  Level  3  in  2003,  Maturity  Level  4  in  2005, 
and  Maturity  Level  5  in  2008. 


■  Mike’s  present  position  is  IDS  Process  Technical 
Director.  He  is  a  certified  CMMI  Instructor.  Mike 
is  a  member  of  the  CMMI  VI  .3  Core  Model  Team, 
the  CMMI  VI. 3  Training  Team,  the  CMMI 
Configuration  Control  Board,  and  the  NDIACMMI 
Working  Group. 
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Do  You  Need  CMMI? 


Recognize  these  symptoms? 

•  Missed  commitments 

-  Late  delivery 

-  Last  minute  crunches 

-  Spiraling  costs 

•  Inadequate  management  visibility 

-  Too  many  surprises 

•  Quality  problems 

-  Too  much  rework 

-  Functions  not  working  correctly 

-  Customer  complaints 

•  Poor  morale 

-  Crisis  atmosphere 

-  High  turnover 

-  Low  productivity 
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Does  the  following  occur? 

•  Poor  planning 

-  Plans  not  realistic  or  followed 

-  Work  is  not  tracked  against  the 
plan;  plans  are  not  adjusted. 

•  Baselines  not  controlled 

-  Inconsistent  requirements 

-  Changes  not  managed 

•  Ineffective  organizational  structure 

-  Functions  not  well  integrated 

-  Designs  not  producible 

•  Unable  to  repeat  successes 

-  Staff  skills  and  knowledge  not 
available  when  needed 

-  Dependent  on  heroic  individuals 


CMMI  Features  Help  Address  Common  Issues 


STRENGTH  TIIKOIGH  1M1LSTKV  &  TECHNOLOGY 


CMMI  Feature 

Description  and  Examples 

Results  Oriented 

•  Industry  best  practices  for  project  planning  and  execution 

•  Performance-driven  measures  for  consistent  outcomes 

Priorities  Based  on 
Business  Value 

•  Investments  and  maturity  prioritized  to  align  with  business  goals 

•  Appraisals  relative  to  model  to  set  direction  (“map  and  compass”) 

Customer  Focus 

•  Validation  of  customer  needs  across  the  project  life  cycle 

•  Manage  product/service  quality  (verification,  validation,  reviews) 

Proactive 

Management 

•  Forward-looking  measurement,  monitoring,  risks,  corrective  action 

•  Management  decisions  based  on  plans,  data,  alternatives 

Flexibility 

•  Adaptable  to  a  variety  of  businesses  (domain,  size,  products) 

•  Non-prescriptive  (required,  expected,  informative  components) 

Business  Process 
Integration 

•  Cross-functional  stakeholder  involvement 

•  Coordinate  various  improvement  strategies  and  methods 
(Lean,  Six  Sigma,  ISO,  Agile,  etc.) 

Continuous 

Learning 

•  Standardized  assets  tailored  for  project  characteristics 

•  Leverage  experience  and  history  across  projects 
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Why  Focus  on  Process? 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


PEOPLE 


The  quality  of  a  system  is  highly  influenced  by  the  quality  of  the 
process  used  to  acquire,  develop,  and  maintain  it. 

•  A  long-standing  premise  in  manufacturing 

•  Good  processes  increase  the  likelihood  of 
successful  projects 

Process  can  enhance  the  capabilities 
of  your  workforce 

•  Work  smarter,  not  just  harder 

•  Leverage  organizational  experience 
and  best  practices 

Process  integrates  technology 
with  resources 

•  Technology,  by  itself,  will  most 
likely  not  be  used  effectively 


TECHNOLOGY 
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What  Is  CMMI? 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


CMMI  is  a  model  representing  a  collection  of  best  practices  proven 
effective  in  industry 

•  A  framework  for  developing,  improving,  and  sustaining  business  performance 

•  Provides  a  process  focus  on  work  activities 

•  Developed  by  industry  (commercial  and  defense),  government,  academia 

CMMI  targets  three  primary  environments: 


•  Development  - 

Engineering  a  product  or  service 

•  Services  - 

Providing  services 

•  Acquisition  - 

Acquiring  products  and  services 

The  CMMI  product  suite 
consists  of: 

•  Models  and  primers 

•  Appraisal  methods 


CMMI-DEV 

\ 

o 

CMMI  Model 

i 

Framework 

i 

to 

< 

n 

Z _ / 


•  Training  courses 


Capability  Maturity  Model  Integration  (CMMI®) 


CMMI  for  Executives 
November  201 1 


7 


What  CMMI  Can  Add  to  Your  Organization 


STRENGTH  TIIKOIGH  IMIISTKV  &  TECHNOLOGV 


•  Integration  of  business  processes  across  functions  based  on  industry 
best  practices 

•  Visible  project  and  organizational  measures  aligned  with 
achievement  of  business  objectives 

•  Commonly  accepted  process  framework  for  inter-company 
coordination  and  competitor  benchmarking 

•  Repeat  project  successes  through  standardization,  tailoring,  and 
capture  of  organizational  process  assets 

•  Avoid  project  performance  issues  through  process  discipline, 
proactive  management,  and  early  stakeholder  engagement 

•  Predictable  project  performance,  with  fewer  surprises 
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CMMI  Model  Overview 


STRENGTH  TIIKOLGH  1M1LSTKV  &  TECHNOLOGY 


Process 

Areas 

Clusters  of  related  practices,  in  several  categories 
•Project  Management  -  planning,  monitoring,  suppliers,  risk,  ... 
•Support  -  CM,  QA,  measurement,  decision  analysis,  ... 

•Process  Management  -  organizational  processes,  training,  ... 
•Engineering  -  requirements,  development,  integration,  ... 

•Services  -  development,  delivery,  transition,  ... 

•Acquisition  -  requirements,  solicitation,  agreements,  ... 

Generic 

Practices 

Enable  process  management,  deployment  and  improvement 

•Plans,  monitoring,  CM,  stakeholders,  objective  evaluation,  ... 

Goals 

Describes  characteristics  for  implemented  processes 

Capability 

Levels 

Achievement  of  process  improvement  within  an  individual 
process  area 

Maturity 

Levels 

Achievement  of  overall  process  improvement  across  a 
predefined  set  of  process  areas  (stages) 
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CMMI  Appraisals 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGY 


Appraisals  compare  organization  and  project  processes 
against  CMMI  models  to  determine  improvement  priorities 

Senior  management’s  role  in  appraisals: 

•  Provide  sponsorship  and  resources 

•  Set  appraisal  scope  and  objectives 

•  Ensure  follow-through  on  appraisal  findings  and  prioritized 
improvement  actions 

CMMI  provides  a  family  of  appraisal  methods,  with  varying 
intent,  confidence  levels,  data  collection,  resources  needed 

•  Flexible  focus:  approach,  deployment,  institutionalization 

•  Rigorous  benchmark  rating  method  (for  maturity  levels) 

•  “Quick  look”  diagnosis  of  process  weaknesses 

Licensed  SEI  partners  deliver  SCAMPISM  appraisal  services  p-  SEI Partner 

•  http://www.sei.cmu.edu/partners/directorv/  CMMI 

Note  that  for  internal  process  improvement,  company- 

developed  and  other  methods  can  be  effective 
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Reasons  You  Should  Adopt  CMMI 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


1.  Increase  customer  satisfaction 

•  Deliver  products  and  services  that  satisfy  user  needs 

•  Deliver  products  and  services  on  time  and  within  budget 

2.  Increase  probability  of  capturing  new  and  repeat  business 

•  Improved  ability  to  meet  commitments 

•  Reduces  customer-perceived  risk  of  award  to  your  organization 

•  Can  be  a  discriminator  relative  to  your  competition 

3.  Increase  profit  through  improved  quality  and  less  rework 

•  Better  predict  actual  costs  through  repeatable  processes 

•  Better  visibility  into  projects  due  to  established  measures  and  analysis  techniques 

•  Significantly  reduce  the  probability  of  problem  programs 

•  Reduce  costs  by  capitalizing  on  organizational  infrastructure,  processes,  training, 
tools  and  early/often  stakeholder  involvement 

4.  Increase  productivity 

•  More  efficiency  through  implementation  of  common  processes,  tools  and  training 

•  Improved  productivity  by  implementing  process  improvement  that  are  directly 
aligned  key  organizational  goals  and  objectives. 

•  Higher  employee  morale  and  less  turnover 


CMMI  for  Executives 
November  201 1 


12 


Benefits  of  CMMI-Based  Process  Improvement 


STRENGTH  TIIKOIGH  INDUSTRY  &  TECHNOLOGY 


Many  companies  cite  performance 
benefits  from  CMMI 


•  Published  in  conferences,  articles, 
papers,  studies,  surveys,  reports 

SEI  collects  quantitative  measures 
of  CMMI  performance  improvement 

•  Technical  reports,  including: 

-  “Performance  Results  of  CMMI- 
Based  Process  Improvement” 
(http://www.sei.cmu.edu/pub/docume 

nts/06 .  reports/pdf/06tr004 .  pdf) 


Performance 

Median 

Category 

Improvement 

Cost 

34% 

Schedule 

50% 

Productivity 

61% 

Quality 

48% 

Customer 

Satisfaction 

14% 

ROI 

4.0  :  1 

CMU/SEI-2006-TR-004. 

Data  from  35  organizations. 
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Industry  Benefits  from  CMMI 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie-  Mellon  University 
Pittsburgh,  PA  15£13 

May  2010 


So  ft  warn  Engineering  Institute  (jurth^Mrtlnn 


Example  measures  reported  by  NDIA  member  companies: 

Defect  repair  effort 

•Defect  repair  hours:  -58%  (ML3  to  ML5) 
•Defect  cost  savings:  -105  hrs  per  defect 
•l&T  hrs/defect:  -24% 

•Hours/KLOC:  -22%  (ML3  to  ML5) 

Defect  density 

•62%  fewer  high-severity  defects  (ML5) 
•Defect  phase  containment:  +240%  (ML5) 
•>85%  defects  removed  prior  to  sys  test 
•Acceptance  test:  <  0.15  defects/KLOC 

Development  cost 

•SW  development  cost:  -28%  (ML3  to  ML5) 
•Potential  project  savings:  $1 .9M-$2.3M 

Productivity 

•Productivity  gain:  +42%  (ML5,  9yrs) 

Cost/schedule 

•Over  6X  less  likely  cost/schedule  impact 

The  new  data  presented  in  this  report  demonstrates  that  effective 
implementation  ol  good  practices  aided  by  use  ol  CMMI  can  improve 
cost,  schedule,  and  quality  performance. 


Benefits  of  CMMI  in  the  Defense  Industry,  Software  Engineering  Institute,  May  2010. 

http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense-lndustry.cfm 
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CMMI  Adoption 


STRENGTH  TIIKOLGH  1M1LSTKV  &  TECHNOLOGY 


CMMI 
appraisals 
are  conducted 
worldwide... 


USA  Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

316 

36.6% 

3040 

92.0% 

468 

54.2% 

191 

5.8% 

80 

9.3% 

73 

2.2% 

864 

100.0% 

3304 

100.0% 

...in  a  wide  range  of  businesses 


...in  small 
and  large 
organizations 
and  projects 


>200, 

18.7% 

101-200, 

15.1% 


<100, 

66.4% 


< 


>2000,  1.6% 
1001-2000,  1.9% 
501-1000,  4.0% 
301-500,  5.4% 
201-300,  5.8% 
101-200,  15.1% 

76-100,  7.6% 
51-75,  12.6% 


26-50,  25.9% 


<25,  20.3% 


Organization  Size  (Employees) 

(4197  organizations  reporting) 


...at  all  levels  of  process  maturity 


Services 

•  Business  Services 

•  Engineering  and 
Management  Services 

•  Health  Services 

•  Other  Services 

Other 

•  Finance,  Insurance,  Real  Estate 

•  Public  Administration/Defense 

•  Transportation,  Communication, 
Utilities 


Contractor  Military/ 


Commercial  for  Military/  Government 
In-House  Government  Agency 


No  Rating  Given 

3.5% 

2.9% 

15.7% 

Initial  (ML1) 

0.4% 

2.0% 

0.7% 

Managed  (ML2) 

20.4% 

27.6% 

43.1% 

Defined  (ML3) 

61.7% 

58.3% 

39.9% 

Quantitatively  Managed  (ML4) 

1.9% 

0.3% 

1.3% 

Optimizing  (ML5) 

4.6% 

5.8% 

2.6% 

(3356  orgs) 

(659  orgs) 

(153  orgs) 

Source:  SEI  Process  Maturity  Profile,  Sept  2011. 
http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 


Manufacturing 

•  Electronic  and  Electric  Equipt 

•  Transportation  Equipment 

•  Instruments  &  Related  Products 

•  Industrial  Machinery 

•  Other  Mfg  Industries 
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Getting  Value  from  CMMI 

Your  Role  as  an  Executive 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


Set  the  vision  and  direction  for  CMMI-based  improvement 

•  Establish  measurable  objectives 

•  Be  a  visible  sponsor  -  set  expectations  for  involvement 

•  Manage  process  improvement  like  a  project 

Provide  resources  and  support 

•  Funding,  staffing,  tools 

•  Choose  the  best  people  to  lead  -  respected  opinion  leaders 

Keep  it  real 

•  Maintain  relentless  focus  on  business  value  and  program  performance 

•  Involve  projects  and  practitioners  for  the  best  ideas 

•  Hold  people  accountable 

•  Track  and  communicate  progress 

•  Recognize  and  reward  achievement 
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The  Effective  Use  of  CMMI® 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


Summary  of  NDIA  industry  position  statements  for  obtaining  best  value 
from  CMMI  investments*: 


1.  Good  processes  increase  the  likelihood  of  achieving  successful  project  performance 

2.  CMMI  is  a  model,  not  a  standard  -  adapt  CMMI  to  your  business  environment, 
resources,  and  objectives 

3.  Focus  on  business  improvement  objectives  -  a  primary  emphasis  on  achieving 
levels  may  not  achieve  significant  benefits  and  may  increase  rather  than  decrease 
costs 

4.  High  maturity  is  a  business  case  -  justify  the  investment:  many  organizations  find 
business  value  in  improving  processes  even  at  lower  CMMI  maturity  levels 

5.  Maturity  level  ratings  are  not  alone  a  predictor  of  project  performance  -  many 
other  factors  can  be  significant  contributors 

6.  Don’t  specify  maturity  levels  in  acquisitions  -  use  CMMI  to  probe  supplier 
capability  ana  process  execution  risks 

7.  Greatest  benefits  of  appraisals  are  from  improvements,  not  evidence  or  ratings  - 

disproportionate  effort  on  appraisal  preparation  risk  can  diminish  business  returns 


“The  Effective  Use  of  CMMI®”,  NDIA  Systems  Engineering  Division,  June  2009. 
http://www.ndia.orq/Divisions/Divisions/SvstemsEnqineerinq/Paqes/CMMI  Working  Group. aspx 
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Want  to  Learn  More  about  CMMI? 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGY 


SEI  CMMI  web  pages: 


What  is  CMMI?  Models 

Conferences  Performance  Results 

FAQs  Background  Information 


CMMI  focus  topics,  guidance,  technical  reports: 


CMMI  and  Agile 
CMMI  in  Small  Settings 
Earned  Value  Management 

Training: 


CMMI  and  Six  Sigma 
CMMI  in  Acquisition 
SW-Only  Organizations 


Process  Improvement  Introduction  to  CMMI 

CMMI  Level  2-3  for  Practitioners  Understanding  High  Maturity 

User  Networks 


SEI  Partner  Network 
Consultants 


Newsgroups,  Blogs,  Wikis 
Conferences 


Questions?  Comments? 


Adoption 

Appraisals 

Contacts 


Product  Line  Practices 
Interpretive  Guidance 
Operations  Organizations 


Intermediate  Concepts  of  CMMI 
SCAMPI  Appraiser  training 


Books,  Periodicals,  Articles 
Asset  Repositories 


Web:  http://www.sei.cmu.edu/cmmi 
Email:  cmmi-comments@sei.cmu.edu 

SEI  Customer  Relations:  (412)  268-5800,  customer-relations@sei.cmu.edu 


CMMI  for  Executives 
November  201 1 


20 


LOCKHEED  M A 


Applying  CMMI-SVC  Process  Areas 
to  CMMI-DEV  Projects 


M.  Lynn  Penn 

Director  Strategic  Process  Engineering 
Integrated  Systems  &  Global  Solutions 
Lockheed  Martin  Corporation 
November,  2011 


LOCKHEED  M  A  R  T  I  ilf  ^ 

Topics  for  Discussion 

Why  look  Beyond  Development? 

Sampling,  of  Specific  Service  Process  Areas  that 
enhance  the  Engineering  Process  Areas 


Benefits  of  Service  PAs  to  Development  Projects 


WHY???? 


LOCKHEED  M A 


•  Expectations  of  the  product  being  developed 

-  Sustainment  expected 

-  Maintenance  phase  expected 
—  Product  Warranties 

—  User  Training 

-  Technology  Refresh 


Product  Development  has  evolved  to  a  Service  System 


Goal  Setting 


LOCKHEED  M A 


•  Production  Goal:  Establish  a  product  that  can 
transition  to  ensure  maintenance  of  the  functionality 
of  the  components  as  well  as  the  total  functionality  of 
the  product: 

-  Plan  for  the  maintainability  of  the  product 

-  Understand  Customer  Needs 

-  Set  appropriate  expectations 

•  Service  Goal:  Plan  for  the  continued  support  of  the 
product  in  use: 

-  Establish  the  Service  System 

-  Understand  Customer  Needs 

-  Set  appropriate  expectations 


LOCKHEED  M A 


Process  Areas  to  Choose 


•  ALA  Carte  Menu,  Please 


LOCKHEED  M A 


Process  Area  Selection 


Although  the  Production  and  Service  Goals  are  similar,  it  is  within 
production  that  the  first  consideration  for  the  service  life  cycle 
needs  to  be  defined,  communicated  and  established.  The  overlap  is 
essential  to  long  term  success. 

•  The  selected  Process  Areas  are  divided  into  three  sets: 

-  SET  1  -  core  process  areas  that  start  implementation  under  a 
production  life-cycle  but  include  service  concerns  so  they  can 
successfully  be  instantiated  into  the  service  life  cycle  (a  sampling), 

-  SET  2  -  those  service  specific  process  areas  (or  goals  within  a  PA)  that 
MUST  be  considered  during  the  production  phase  and  NOT  wait  for 
the  Services  life-cycle  to  commence,  and 

-  SET  3  -  those  process  areas  (or  goals  within  a  PA)  that  will  be  initiated 
and  implemented  during  the  Services  life-cycle  (SET  1  and  SET  2  will 
continue  to  evolve  during  the  Services  life-cycle). 


Core  process  areas  that  start  implementation 
under  a  production  life-cycle  but  include 
service  concerns  so  they  can  successfully  be 
instantiated  into  the  service  life  cycle 

-  Risk  Management  (RSKM) 

-  Requirements  Management  (REQM) 


ONLY  A  SAMPLE  -  OTHERS  MAY  APPLY 
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Risk  Management  (RSKM) 
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SG  1  -  Prepare  for  Risk  Management 

•  Identify  Risk  Sources  and  Categories 

-  These  should  include  both  sources  from  the  development  and 
operational  life  cycles 

•  Define  Risk  parameters 

-  Evaluation  of  development  risks  should  include  their  potential  risks 
into  the  operational  service  of  the  product 

-  Likelihood  of  occurrence  needs  to  apply  to  both  within  development 
and  during  operation 

•  Establish  a  Risk  Management  Strategy 

-  Strategy  needs  to  apply  to  decisions  made  during  development  that 
could  identify  a  risk  in  operation 

-  Broaden  the  strategy  to  be  forward  looking 

•  Decisions  made  during  development  could  identify  a  risk  during  operation 


Risk  Management  (RSKM) 

(continued) 

SG  2  -  Identify  and  Analyze  Risks 

•  Identify  Development  and  Operational  Risks 

—  Identification  must  include  both  life  cycles 

—  Identification  of  a  development  risk  could  identify 
additional  operational  risks 

•  Analyze  Risks 

—  Analysis  of  a  development  risk  could  identify  an  additional 
operational  risk 

•  Mitigation  Plans 

—  Mitigation  plans  should  be  seamless: 

•  Including  steps  in  development  to  mitigate  operational  risk 

•  An  operational  risk  may  lead  to  additional  development 
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Requirements  Management  (REQM) 


~7 


SGI  -  Manage  Requirements 

•  Understand  Requirements 

-  During  development  requirements  must  be  prioritized  to  gain  insight  into  the 
customer's  priority  needs 

-  This  is  the  first  step  in  understanding  the  service  requirements  pertaining  to 
functionality 

•  Manage  Requirements 

-  As  requirements  change,  analysis  as  to  the  service  capabilities  and  delivery 
must  occur 

•  Bidirectional  Traceability 

-  Critical  to  ensure  that  the  requirements  do  indeed  meet  the  service 
requirements 

•  Identify  Inconsistencies 

—  Essential  to  not  only  look  at  inconsistencies  when  delivering  the  product  but 
inconsistencies  in  the  final  delivery  of  service  on  the  product 


Those  service  specific  process  areas  (or  goals 
within  a  PA)  that  MUST  be  considered  during 
the  production  phase  and  NOT  wait  for  the 
Services  life-cycle  to  commence 

-  Service  System  Transition  (SST) 

-  Strategic  Service  Management  (STSM) 

-  Capacity  and  Availability  Management  (CAM) 

-  Service  System  Development  (SSD) 

-  Service  Continuity  (SCON) 


Service  System  Transition  (SST) 


SG  1  -  Prepare  for  Service  System  Transition 

•  Analyze  and  Develop  Transition  plans 

-  Include  this  analysis  during  both  Requirements  Management 
and  Product  Integration  during  Development 

-  Requirements/  Functions/  Tools/  Components  must  all  factor 
into  transitioning  the  system 

-  Analyze  Warranties  and  Technology/  Hardware  refresh  needs 
and  relate  those  needs  to  the  production  phase 

•  Prepare  Stakeholders 

-  With  considerations  around  warranties  and  refresh  -  production 
can  set  the  expectation  for  service  capabilities 

-  Customers  should  buy  the  product  knowing  what  services  are 
provided  in  association  with  the  product 


Strategic  Service  Management  (STSM) 


SG  1-  Establish  Strategic  Needs  and  Plans  for  Standard  Services 

•  Understand  the  "life  of  product" 

—  Define  the  typical  length  of  time  the  product  will  be  in  service 

•  Define  what  is  and  is  not  applicable  to  "standard  service" 

-  Define  the  goal  of  the  organization  to  commit  to  servicing  this  product 

•  What  will  be  covered  and  what  will  not  (warranties,  maintenance  plans,  extended 
warranty  options...) 

•  Will  servicing  be  supplied  with  resources  from  within  or  will  separate  components  be 
covered  by  suppliers 

•  Achieve  organizational  and  customer  agreements  on  what  is  and  is  not 
standard  services 

•  Define  the  organizational  strategy  for  service  implementation  and 
commitment 

—  Understand  what  this  means  for  the  business  future 

-  Does  it  fit  with  organizational  business  goals 

•  Needs  to  begin  during  production  in  order  to  establish  appropriate 
communication  and  paths  going  forward  from  production  to  service 


Strategic  Service  Management  (STSM) 

(continued) 

SG  2  -  Establish  Standard  Service 

•  Once  we  have  identified  the  standard  services 
that  will  be  provided  either  internally  or  via 
suppliers: 

-  Define  service  levels 

•  These  should  include  response  time/  acceptable  down 
time/  support  via  phone/  support  via  visits/  support  via 
mailing  or  visiting  a  central  location 

-  Define  acceptable  variation 

•  Pricing/  resources/ Timetable  for  upgrades 


Capacity  and  Availability  Management  ‘ 

(CAM) 


SG  1  -  Prepare  for  Capacity  and  Availability  Management 

•  Strategically  identify  what  is  needed  to  maintain  and  support  the 
product  when  required 

-  This  will  influence  the  components  that  are  acquired  and  integrated 
into  the  final  product 

-  Establish  a  commitment  for  the  release  of  the  product  into  the  service 
domain 

—  Identify  any  overlap  on  production  fixes  versus  normal  servicing  and 
maintenance 

•  Imperative  that  commitments  on  service  maintenance  and 
replacement  be  understood  as  part  of  production  choosing 
suppliers  of  individual  components 

•  Identify  measurements  that  the  system/  production  vendors  can 
commit  to  as  they  evolve  or  transition  into  service  life  cycle 
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Capacity  and  Availability  Management  (CAM) 

(continued) 
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SG  2  -  Monitor  and  Analyze  Capacity  and 
Availability 

•  Identification  of  Planned  Capacity  and 
Planned  Availability  should  be  established 
during  the  Production  phase  so  that 
monitoring  of  capacity  and  availability  can 
occur  during  the  services  life  cycle 

-  Reports  on  the  monitoring  will  be  generated 
during  the  service  life  cycle 


Service  System  Development  (SSD) 


SG  1  -  Develop  and  Analyze  Stakeholder  Requirements 

•  Identifying  stakeholder  requirements  for  the  product  after 
it  is  in  use  is  critical  to  developing  the  service  requirements. 
As  production  is  going  on  each  component  may  have  a 
separate  stakeholder  requirement  or  expectation  for 
longevity/  maintenance/  warranty 

•  Depending  on  the  complexity  of  the  product  itself  this 
exercise  needs  to  be  initiating  very  early  in  the  production 
cycle 

•  Commitment  and  agreement  on  these  expectations  needs 
to  be  defined  -  if  these  are  not  validated  -  the  product  may 
not  sell! 


Service  System  Development  (SSD) 

(continued) 

SG  2  -  Develop  Service  Systems 

•  Service  system  solutions  should  be  evaluated  during 
production 

—  Critical  during  Technical  Solutions  that  this  be  coupled 

•  Trade  studies  on  tools/  component  selection 

•  Allocation  of  requirements  to  functions  to  set  service  expectations 

•  Developing  the  production  design  to  ensure  service  system 
capability 

•  During  Product  Integration  consideration  to  service 
system  may  come  to  play  as  the  system  is  built  - 
particular  reference  to  functional  deliveries  (Agile 
environments) 


Service  Continuity  (SCON) 


SGI  -  Identify  Essential  Service  Dependencies 

•  In  keeping  with  both  Requirements  Development  and 
during  Requirements  Management  it  is  critical  that  the 
customer  priorities  on  functionality  be  agreed  to 

•  Prioritization  of  functions  may  be  an  input  to  Risk 
Management,  Decision  Analysis  and  Resolution,  and 
Technical  Solutions  as  the  product  is  being  developed 

•  As  requirements/  functions  are  being  developed, 
management  will  have  an  insight  into  the  resources 
needed  into  services  (number  of  people/  skills  and 
knowledge  of  those  individuals/  budget/  timeline) 


Service  Continuity  (SCON) 

(continued) 

SG2  -  Prepare  for  Service  Continuity 

•  Continuity  plans  and  training  should  be  accomplished 
during  production 

—  Depending  on  expectations  of  the  plan  -  production/ 
product  design/  product  integration  could  be  affected 

SG3  -  Verify  and  Validate  the  Service  Continuity  Plan 

•  An  actual  dry-run  of  the  continuity  plan  would  be 
worthwhile  during  production  since  results/  good  and 
bad  should  go  back  into  the  production 

—  If  done  during  production  continuity  risks  can  be  mitigated 
by  a  change  in  the  product  configuration  -  a  possibility 


Those  process  areas  (or  goals  within  a  PA)  that  will 
be  initiated  and  implemented  during  the  Services 
life-cycle  (SET  1  and  SET  2  will  continue  to  evolve 
during  the  Services  life-cycle) 

-  Service  Delivery  (SD) 

-  Incident  Resolution  and  Prevention  (IRP) 

-  Service  System  Transition  (SST) 

-  Service  System  Development  (SSD) 


Service  Delivery  (SD) 


SGI  -  Establish  Service  Agreements 

•  This  activity  since  customer  expectations  have 
been  addressed  during  production  should  be 
understood. 

—  In  lieu  of  reviewing  "existing"  Service  Agreements  - 
can  evaluate  those  commitments  that  were  planned 
for  during  production 

•  Establishing  the  service  agreement  with 
customers  is  more  clearly  negotiated  since 
requirements  are  aligned  to  the  service 
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Service  Delivery  (SD) 
(continued) 


SG2  -  Prepare  for  Service  Delivery 

•  Many  of  the  inputs  to  these  systems  and  plans  have  been  identified  during 
the  production  phase,  this  step  within  the  service  life  cycle  is  more  a 
formalization  of  approaches  and  management  systems  around  the  service 

•  Preparation  will  be  based  on  the  production  expectations  and  service 
capabilities  established  again  during  production 


SG3  -  Deliver  Services 

•  The  product  has  been  developed  with  the  service  expectations  (internal 
and  external)  in  site  -  service  delivery  should  be  available  with  a  shorter 
start  up  lag  time 

•  Management  systems  should  be  easier  to  identify  and  start  up  (customer 
expectations  are  understood  and  internal  budgeting  for  resources  have 
been  identified) 

•  Operations  should  be  focused  back  to  agreements  and  requirements  that 
were  identified  during  production 


Incident  Resolution  and  Prevention' 


(IRP) 

SGI  -  Prepare  for  Incident  Resolution  and  Prevention 

•  Incident  definition  needs  to  be  established. 

-  Definition  should  be  based  on  expectations  set  during 
production 

—  Differentiation  of  request  versus  incident  -  similar  to  any 
service  start  up 

•  Differentiation  should  be  defined  within  production  context 

•  Establish  Incident  Management  System 

—  Resources  and  tools  assigned  to  the  management  system 
based  on  analysis  during  production  -  Risk  Management/ 
Product  Integration  alignment 


Incident  Resolution  and  Prevention  (IRP) 

(continued) 

SG2  -  Identify,  Control,  and  Address  Incidents 

•  Periodically  as  well  as  event  drive  -  accommodate  and 
plan  to  analyze  incidents  to  avoid  recurrence 

•  Critical  to  keep  communication  open  with  customers  - 
integrated  with  Service  System  Transition  when 
applicable 

•  May  require  returning  to  production  phase  if 
component  selected  during  production  is  not 
repairable/  if  functionality  is  in  question 

—  This  areas  could  directly  result  in  production  issues  being 
identified 


Incident  Resolution  and  Prevention  (IRPp 

(continued) 

SG3  -  Define  Approaches  to  Address  Selected 
Incidents 

•  Identification  of  systemic  incidents  or 
incidents  whose  solution  may  not  be 
appropriate  for  a  component  transition 

-  Identification  of  these  incidents  may  require 
customer  communication  for  awareness/ 
workarounds/  loss  of  functionality  etc 


Service  System  Transition  (SST) 


SG2  -  Deploy  the  Service  System 

•  The  plans  should  be  in  place  for  scheduled 
transition  activities  therefore  deploying  a  refresh/ 
warranty  component  etc  is  already  in  place 

-  Unplanned  transitions  need  to  address  SGI  again 
within  the  operational  phase 

•  Assessing  the  transition 

-  This  activity  may  change  the  plan  or  cause  a  re-entry 
to  production  to  reevaluate  the  selection  of  that 
component 


Service  System  Development  (SSD) 


SG  3  -  Verify  and  validate  Service  Systems 

•  Actual  verification  of  the  service  system  will  not 
occur  until  the  operational  phase  although  the 
actual  integration  of  the  components  and  the 
service  design  can  be  accomplished  during 
production  phase 

•  Peer  reviews  should  include  production  staff  as 
service  staff  should  have  been  included  in 
verification  activities  during  production 

-  Both  service  and  production  are  stakeholders  in  the 
product  solution  set 


Benefits,  Benefits,  Benefits... 


Product  Integration  coupled  with  Service  System  Transition 

-  Internally  -  impact  awareness  for  management/  configuration 
management/  test/  quality 

-  Externally  -  confidence  system  will  continue  to  operate 

Adding  Service  Continuity  (a.k.a.  System  Continuity) 

-  Customer  and  users  MUST  identify  those  requirements 
associated  with  critical  functions 

-  Test  Engineers  focus  on  system  failure  as  well  as  individual 
component  failures  to  test  for  continuous  critical  functionality 

Risk  Management  coupled  with  Capacity  and  Availability 

-  Expands  the  focus  from  internal  development  to  external 
operations 

-  Bridges  project  evolution  from  production  to  sustainment 


Moral  of  the  story 


It  is  essential  in  this  economy  and  environment  that 

production  needs  to  keep  in  mind  that  service  of  a  product 
is  a  critical  issue.  The  operational  life  cycle  will  indeed  last 
a  "life  time"  and  it  is  imperative  to  give  consideration  to  the 
requirements/  resources/  expectations  of  this  phase  as  the 
product  is  being  built. 


•  Encourage  the  use  of  CMMI  -  SVC  as  an  extension  to 
CMMI-  DEV 

•  Proactively  selecting  process  areas  regardless  of  the 
constellation  to  meet  customer  needs 

-  Multi-dimensional  view  of  quality 

-  Production  and  Operational  life  cycles  supported 
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...And  the  Walls  Come 
Tumbling  Down 


Questions?? 
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Systems  Center 
PACIFIC 


A  LEAN  and  RACI  Approach  to  CMMI 
for  Services  (CMMI-SVC) 


Ahn  Nuzen  &  David  Dayton 

SPAWAR  SYSTEMS  CENTER  PACIFIC  (SSC-Pacific) 

US  NAVY 


A 


We  are  the  Navy  experts  in  delivery  and  sustainment  of  C4ISR  systems 


SFMAR 

y 

Systems  Center 
PACIFIC 


OUTLINE 


T  Background  about  SPAWAR  Systems  Center  Pacific 
T  Why  this  approach? 

T  What  is  RACI? 

T  How  to  populate  RACI? 

T  Mapping  RACI  into  CMMI  SVC  Specific  Practices 


A 


We  are  the  Navy  experts  in  delivery  and  sustainment  of  C4ISR  systems 
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Systems  Center 
PACIFIC 


Space  &  Naval  Warfare  Systems  Center 
Pacific  -  SSC  Pacific 


T  DOD  US  NAVY  Organization 

■  4000  +  Scientists  &  Engineers 

■  Located  in  San  Diego,  and  throughout  the  Globe 


A 


We  are  the  Navy  experts  in  delivery  and  sustainment  of  C4ISR  systems 
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system  center  Mission  ~  Information  Dominance 


Design,  Build,  and  Sustain 
C4ISR  Information 
Dominance  Systems 


* 

'l 


(Radar,  Networks,  Command 
and  Control,  Crypto  Devices, 
Satellites  communications, 
Submarines  Electronic 
Systems,  etc... ) 


A 


We  are  the  Navy  experts  in  delivery  and  sustainment  of  C4ISR  systems 


SPWAR 

Systems  Engineering  for  Mission  Success 


Reliability 


Availability 


Maintainability 


A 
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Systems  Center  SSC  Pacific  CMMI  Timeline 


A 
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SFMAR 

J?L  SSC  Pacific  CMMI  SERVICES 


T  Diversity  of  projects  involved  in  systems  engineering 
services,  Research  and  Development  (R&D),  logistic, 
maintenance,  sustainment,  etc. 

T  Some  are  non-product-oriented  projects  do  not  maximize 
the  benefits  of  CMMI-DEV  initiative 

T  2009  adopted  CMMI-SVC,  version  1.2  to  improve  the 
projects’  performance  and  quality  for  non-product-oriented 
projects 


A 
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"WM 

sy,Jcr,  Approach  to  gather  Artifacts  for  CMMI? 

▼  Traditional 

■  Mapping  business  process  into  the  CMMI  Model 

-  Business  must  learn,  and  speak  CMMI 

-  Steep  learning  curve 

-  Time  consuming 

▼  Lean 

■  Begin  with  the  existing  “as-is"  business  process  mapping 

-  Ask  the  questions  to  gather  the  artifacts: 

1.  What  process  is  been  performed? 

2.  How  is  the  process  being  performed? 

3.  Why  the  process  is  been  performed? 

4.  Who  are  the  process  role-players?... 


A 
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SPWAR 

Sf~,r  Example  of  “as-is”  Business  Process 


Functions 


Processes 


Functionaries  & 
Tools 


Drivers,  Purpose 
Requirement 
REFERENCES  & 
reasons  (e.g., 
Best  practices) 


Work  Plan  Task  Management 

Pre-Planning 

Funds 

Management 

Design  Review 

Cost  Est  V&V 

POA&M  Development 

Actual  Cost  MRN&C 

SKED  De-confliction 

Biz  Mgt  Grp 

CETracker 

CETracker  Review 

ERP 

SPIDER  Review 

EVM  Reports 

Performance 

Agreements 

FPY  Hot  Wash 
Review 

Minimize  Est-to-Act 
Cost  Var 

Performance 

Agreements 

GFE/GFM 

Timelines 

A 
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Continuous  Process  Improvement 
sZfL  A  Concept  of  Operations  (CONOPS) 


SFMAR 

y 

Systems  Center 
PACIFIC 


What  is  RACI  Matrix? 


T  Pronounced  “racy”  or  “rack-y” 

T  A.K.A.  RACI-ARCI  Matrix 

■  Although  ARCI  more  accurately  reflects  left-to-right  hierarchical  roles,  RACI  seems  to  be 
the  acronym  most  widely  used  in  industry 

T  Useful  in  project  management 

■  WHO  does  WHAT,  WHEN 

T  Useful  in  Continuous  Process  Improvement  (CPI)  and  Lean/Six 
Sigma;  facilitates  “as-is"  to  “to-be”  organization  planning  and  process 
management 

■  WHO  does  WHAT,  WHEN,  WHY 

■  CTQ  Outcomes;  Time  &  Cost  per  process  step;  etc. 

▼  Used  to  assign  or  describe  cross-functional  process-related  roles 

▼  Used  to  develop  business  rules  for  process  streamlining  & 
automation 


A 
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Basic  RACI  Matrix 


Responsible 


Those  who  do  work  to  achieve  the  task.  The  role  of  Responsible  includes  support, 
which  is  to  provide  resources  to  complete  the  task.  Responsible  is  about  doing  the 
work;  several,  or  all,  may  share  responsibility.  Responsible  is  linked  to  the 
function(s)  assigned  to  execute  a  particular  activity.  The  degree  of  responsibility 
is  determined  by  the  Accountable  person. 


Accountable 


(Also  Approver  or  final  Approving  authority)  Those  who  are  ultimately  accountable 
for  the  correct  and  thorough  completion  of  the  task.  Accountable  is  the  one  to 
whom  “R(s)”  are  accountable.  There  must  be  only  one  A  specified  for  each  task. 
The  role  of  Accountable  may  include  Responsible;  in  other  words,  it  is  not  unusual 
that  the  one  who  is  Accountable  for  a  task  is  also  Responsible  to  do  the  work  to 
achieve  the  task.  Accountability  cannot  be  delegated. 


Consulted 


Informed 


Those  who  must  be  ‘consulted’  before  decision  or  activity  is  finalized.  A  two-way 
communication  (a  negotiated  consensus). 


Those  who  must  be  notified  about  the  completion  or  output  of  decision  or  activity. 
A  one-way  communication. 


A 
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-  RACI  Matrix  Nautical  Example 

SysPAc/Ficter  (Basic  scheme) 


RACI 


A  tool  for  assigning  cross-functional  roles. 


Nautical  Example 


R  RESPONSIBLE 
A  ACCOUNTABLE 
C  CONSULTED 
I  KEEP  INFORMED 


Responsible  is  about  doing  the  work;  several,  or  all,  may  share  responsibility. 
Accountable  is  defined  as  'the  buck  stops  here';  only  one  individual  can  be  accountable. 


MISSION/CHARTER  TASK 

"To  Ready  &  Deploy  the  Ship" 

CAPTAIN 

NAVIGATOR 

1/2  OFFICER 

CHIEF  ENGINEER 

PURSER 

STORES  PROVISIONS 

PORT  AUTHORITY 

MILESTONE /TASK 

1 

CHART  ROUTE 

C 

R/A 

1 

1 

2 

ORDER  PROVISIONS 

C 

C 

R/A 

3 

ORDER  FUEL 

C 

A 

R 

4 

GAIN  APPROVAL  TO  LEAVE 

A 

R 

R 

R 

5 

SET  SAIL 

A 

C 

R 

1 

6 

TAKE  CONTROL  FROM  PILOT 

R/A 

1 

1 

Activities 


Functions 

are  assigned  roles  in  the  process. 


include  all  the  tasks  that  need  to  be  completed,  as  well  as  the  decisions  that  need  to  be  made. 
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SFMAR 

s7£rr  Mapping  RACI  -  CMMI  Process  Areas 


A  B  C  D  E  F  G  H  W  AD  AL  AR 


CPI  LSS  RACI  MATRIX 


11th  Annual  CMMI  Technology  Conference 


PROCESSES 

PROCESS  STEPS 

Executed  by  Primary  &  Secondary  /  supporting  Competencies  (Roles)  — > 

(As  possible,  start  with  an  ACTION  VERB;  e.g.,  Conduct.  Review;  Develop;  Provide, 
Analyze,  Test,  Validate,  Verify,  Certify,  Approve,  Deliver,  etc.) 

CMMI/PA 

PM/CHENG 

Softvuare  Team 

Subject  Matter 

Expert 

"C 

S 

1 

£ 

Pr 

sT 

52  53  54  55 

T 

1 

1 

T  T  T  T  T  T 

T 

T 

T 

T 

T 

Soil 

.ware  Support  Activity  (SSA) 

REQM  1 . 1 

MA1.4 

REQM 

MA1.4 

MA  1 . 1 

MA1.2 

1.  Review  request  for  update/change  to  established  system  configuration. 

A 

R 

c 

2-  Determine  cost  and  feasibility  of  implementing  the  change. 

l/C 

R 

3-  Review  applicable  requirements  documentation 

R 

c 

4-  Determine  if  an  available  COTS  solution  exists. 

l/C 

R 

5-  Select  the  best  solution  condidate(s). 

l/C 

R 

c 

6-De 

termine  if  the  cost  of  implementation  over  S50K? 

l/C 

R 

6  a-  If  NO.  then  go  to  step  8 

6b- If  Yes.  then  next  step. 

7-  Complete  a  white  paper  for  submission  with  the  Preliminary  ECP/ECR 

l/C 

R 

3-  Request  appropriate  forms  from  the  appropriate  CM  analyst. 

R 

9-  Copy  the  SSA  templates  into  a  new  folder  created  specifically  for  the 
software  ECP/ECR  in  "Working  Software  ECPs/ECRs". 

R 

10-  Fill  in  the  appropriate  form  after  the  SSA  ECP/ECR  forms  have  been  filled 
in  by  a  SSA  rep. 

1 

C 

R 

A 
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T 

Systems  Center 
PACIFIC 


Project-to-Process-Agent  Interactions: 


Project 

Activities 


Maintain 


Coaching  and  collaboration 
for  potential  process  updates 


Process  Agent 
Activities 


Monitor 


Maintain 


Map 


RAC  I  Matrix 


PIIDs  of  CMMI  compliance 


SFMAR 

sJL.  RACI  ~  CMMI  Workbook 


T  Mapping  CMMI  practices  to  RACI  matrix 

■  Completed  by  the  Process  Group  (PG)  who  are  experts  in  CMMI 
structure  and  content 

-  CMMI  practices  (as  mapped)  cited  within  RACI  matrix  first 

-  Can  be  easily  identified  and  initially  cited  in  broad 
percentages  within  project  instantiations  (work  areas) 

-  Next  CMMI  table  of  process  implementation  indicators  (Pll)  is 
populated  with  RACI  content 

-  links  to  data  in  the  RACI  matrix  content 

-  Identify  model  gaps  within  RACI  matrix  and  CMMI  table  and 
provide  feedback  to  project  staff  in  broad  percentages  first 

-  Additional  detail  may  or  may  not  be  deemed  relevant 
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SFMAR 

sJL.  RACI  ~  CMMI  Workbook 


T  Mapping  CMMI  practices  to  RACI  matrix 

■  Gaps  may  be  with  respect  to  the  RACI  template  itself  or  its 
content 

■  Gaps  are  explained  to  staff  in  project  terminology  based  on 
project  needs  and  objectives 

T  “Main  product”  is  an  improved  organization  with  evidences 
provided  in  updated  and  improved  RACI  matrices 

T  “By  product”  is  a  populated  CMMI  Process  Improvement 
Indicator  (Pll)  table  with  current  artifacts  of  implementation 
organized  and  current  for  appraisal  purposes 
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yw 

Systems  Center  Overarching  RACI  Matrix 


T  There  will  be  gaps  from  RACI  because  of  its  as-is  business 
nature  and  content 

■  Gaps  are  to  be  analyzed  for  project  relevance,  not  automatically 
addressed  or  even  assigned  as  action  items  without  a  “business 
case”  first 

■  If  gaps  result  in  business-case  actions  to  be  addressed,  there  will 
be  facilitative  communication  to  the  project  using  their  own  pre¬ 
existing  (non-CMMI)  work  terminology 
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Systems  Center 

PACIFIC 


Thank  you !!! 


ahn.nuzen@naw.mil 

david.a.dayton@navy.mil 
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*  Definition  of  Scrum 

*  Agile  Principles 

*  Definition  of  CMMI 

*  Similarities  and  Differences 

*  CMMI  and  Scrum  Mapping 

*  How  About  Other  Components  of  Level  2? 

*  How  About  Level  3? 

*  Summary 


Full  comparison  at:  http://www.processgroup.com/pgpostmar09.pdf 
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Definition  of  Scrum 


•  Scrum  is  a  pre-defined  development  lifecycle  based  on  Agile 
principles. 


O) 

CD 

c 

CD 

■  — 

c 

o 

c 

c 

c 

CD 

CD 

c 

CD 

OG 

Q_ 

■*— • 

0 

CO 

CD 

0 

CL 

o 

~o 

-•— > 
c 

o 

Q_ 

L_ 

CL 

0 

DC 

CO 

< /) 

CD 

C 

< 


Team 

defined 

phases 


£ 


Potentially  Deliver 


0 

> 

£ 

-i— > 

o 

0 

0 

c 

CD 

'0 

0 

■0 

o 

175 

,0 

> 

0 

oc 

CL 
0 
o 
-1— > 

0 

Q 

O 

1— 

c 

" 

0 

QC 

cl 

-i— > 

CO 

c 

Q. 

CO 

D) 

O 

CD 

c 

O 

0 

GO 

'c 

c 

0 

Q_ 

0 

_c 

> 

0 

CL 

□c 

CO 

( n 
'0 

CD 

C 

< 


£ 


Potentially  Deliver 


0 

> 

£ 

"•4— > 

O 

0 

0 

c 

CD 

'0 

0 

"D 

o 

175 

,0 

> 

0 

DC 

CL 

0 

O 

s_ 

*i— > 

0 

Q 

O 

\— 

c 

" 

0 

C£ 

Q_ 

CO 

c 

CL 

CO 

CD 

O 

CD 

c 

o 

0 

00 

'c 

c 

0 

c 

£ 

0 

Q_ 

■4— • 

c 

'c 

"c 

> 

£ 

0 

CL 

< 

oc 

CO 

>» 

<0 

■a 


>* 

■a 


2-4  week  Sprint 


2-4  week  Sprint 
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Agile  Principles -i 

f  Our  highest  priority  is  to  satisfy  the  customer  through  early  and 
continuous  delivery  of  valuable  software. 

*  Welcome  changing  requirements,  even  late  in  development.  Agile 
processes  harness  change  for  the  customer's  competitive 
advantage. 

^  Deliver  working  software  frequently,  from  a  couple  of  weeks  to  a 
couple  of  months,  with  a  preference  to  the  shorter  timescale. 

»  Business  people  and  developers  must  work  together  daily 
throughout  the  project. 

}  Build  projects  around  motivated  individuals.  Give  them  the 
environment  and  support  they  need,  and  trust  them  to  get  the  job 
done. 


CMMI  compatible 


Source:  http://agilemanifesto.org/ 
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Agile  Principles -2 

The  most  efficient  and  effective  method  of  conveying  information 
to  and  within  a  development  team  is  face-to-face  conversation. 

Working  software  is  the  primary  measure  of  progress. 

Agile  processes  promote  sustainable  development.  The  sponsors, 
developers,  and  users  should  be  able  to  maintain  a  constant  pace 
indefinitely. 

Continuous  attention  to  technical  excellence  and  good  design 
enhances  agility. 

Simplicity  -  the  art  of  maximizing  the  amount  of  work  not  done-is 
essential. 

The  best  architectures,  requirements,  and  designs  emerge  from 
self-organizing  teams. 

At  regular  intervals,  the  team  reflects  on  how  to  become  more 
effective,  then  tunes  and  adjusts  its  behavior  accordingly. 
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Level 

Focus 

Process  Areas 

Quality 

5  Optimizing 

Continuous 

Process 

Improvement 

Causal  Analysis  and  Resolution 

Organizational  Performance  Management 

Productivity 

/ 

4  Quantitatively 
Managed 

Quantitative 

Management 

Organizational  Process  Performance 
Quantitative  Project  Management 

3  Defined 

Process 

Standardization 

Decision  Analysis  and  Resolution 

Integrated  Project  Management 

Organizational  Process  Definition 
Organizational  Process  Focus 

Org  anizatio  n  ai  Tra  i  n  i  ng 

Product  Integration 

Requirements  Development 

Risk  Management 

Technical  Solution 

Validation 

Verification 

/ 

2  Managed 

Basic  Project 
Management 

Configuration  Management 

Measurement  and  Analysis 

Project  Monitoring  and  Control 

Project  Planning 

Process  and  Product  Quality  Assurance 
Requirements  Management 

Supplier  Agreement  Management 

1  Risk 

1  Initial 

Rework 

Model  Source:  http://www.sei.cmu.edu/cmmi/tools/ 
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CMMI  is  a  collection 
of  practices  that  an 
organization 
(software,  hardware 
and  IT)  can  adopt  to 
improve  its 
performance. 

Maturity  Level  2 
Process  Areas  focus 
on  change  and 
project  management. 

Maturity  Level  3 
focuses  on 
engineering  skills, 
advanced  project 
management  and 
organizational 
learning. 
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Visibility  Into  the  Process 

Level  1 


THE 

PROCESS 

GROUP 


•  Process  is  an  amorphous  entity 

•  Visibility  into  the  project’ s  process  is  limited 

•  Difficult  to  establish  the  status  of  the  project’ s  progress 
and  activities 


©  Copyright  2004-2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  1.7 


7 


THE 

PROCESS 

GROUP _ 

Visibility  Into  the  Process 

Level  2 


IN  | - 1 

- ►  - ► 

- a 

*  Customer  requirements  and  work  products  are  controlled 

*  Basic  project  management  practices  have  been  established 

*  Management  controls  allow  visibility  into  the  project  on 
defined  occasions 

*  Management  reacts  to  problems  as  they  occur 


Major  process  phases 
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Level  3 
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i  : 

]  z 


o  o  o 


OUT 


i - 1  [ 

I - 1  c 


]  c 


o  o 


o  o  o 


o 


*  Tasks  in  the  project’ s  defined  process  are  visible 

*  Accurate  and  rapid  status  updates  are  available 

*  Management  proactively  prepares  for  risks  that  may  arise 
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Similarities  and  Differences 


In  Scrum? 


5  Optimizing 

Continuous 

Process 

Improvement 

Causal  Analysis  and  Resolution  l||j||p 

Organizational  Performance  Management 

4  Quantitatively 
Managed 

Quantitative 

Management 

Organizational  Process  Performance  ^H1 

Quantitative-  Project  Management  Hi 

3  Defined 

Process 

Standardization 

Decision  Analysis  and  Resolution  ^HJ 

Integrated  Project  Management  ^Hl 

Organizational  Process  Definition  H :  1 

Organizational  Process  Focus  H] 

Organizational  Training  HI 

Product  Integration  HI 

Requirements  Development  HI 

Risk  Management  HI 

Technical  Solution  HI 

Validation  HI 

Verification  II 

2  Managed 

Basic  Project 
Management 

555 

Configuration Managcpu^^^To  \ 
MeaspiiBo^^T  yjg\i 

^cesrid  Product  Quality  Assurance 

Requirements  Management  [1 

Supplier  Agreement  Management  |jH 

1 1nitial 

inn 

No 


Level  3  coverage  - 
very  dependent  on 
how  YOU  define 
the  phases 


Some  requirements 
Some  design 
Coding 
Some  test 

Some  lessons  learned 

Most  Requirements  Management 
Most  Project  Planning 
Most  Project  Monitoring/Control 
Most  Measurement  Analysis  (effort 
and  progress) 
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CMMI  and  Scrum  Mapping 
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REQM 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Develop  an  understanding  with 
the  requirements  providers  on 
the  meaning  ofthe 
requirements. 

•  Review  of  Product  Backlog  (requirements)  with 
Product  Owner  and  team. 

SP  1.2 

Obtain  commitment  to  the 
requirements  from  the  project 
participants. 

•  Release  Planning  and  Sprint  Planning  sessions  that 
seek  team  member  commitment. 

SP  1.3 

Manage  changes  to  the 
requirements  as  they  evolve 
during  the  project. 

•  Add  requirements  changes  to  the  Product  Backlog. 

•  Manage  changes  in  the  next  Sprint  Planning 
meeting. 

SP  1.5 

Identify  inconsistencies  between 
the  project  plans  and  work 
products  and  the  requirements. 

•  Daily  Standup  Meeting  to  identify  issues. 

•  Release  planning  and  Sprint  Planning  sessions  to 
address  inconsistencies. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining . 

•  Release  Burndown  chart  that  tracks  story  points  that 
have  been  completed.  This  shows  how  much  ofthe 
product  functionality  is  left  to  complete. 

No  traceability  in  Scrum 

[SP  1.4  Maintain  bidirectional  traceability  among  the  requirements  and  work  products] 
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pp 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Establish  a  top-level  work 
breakdown  structure  (WBS)  to 
estimate  the  scope  of  the  project. 

•  The  standard  tasks  used  in  a  Scrum  process 
combined  with  specific  project  tasks  (Scrum 
Backlog). 

SP  1.2 

Establish  and  maintain  estimates 
of  the  attributes  of  the  work 
products  and  tasks. 

•  Story  Points,  used  to  estimate  the  difficulty  (or 
relative  size)  of  a  Story  (requirement). 

SP  1.3 

Define  the  project  life-cycle 
phases  upon  which  to  scope  the 
planning  effort. 

•  The  Scrum  process. 

SP  1.4 

Estimate  the  project  effort  and 
cost  for  the  work  products  and 
tasks  based  on  estimation 
rationale. 

•  Scrum  Ideal  Time  estimate  (similar  to  billable 
hours  or  Full-time  Equivalents). 

SP  2.1 

Establish  and  maintain  the 
project’s  budget  and  schedule  . 

•  Scrum  estimates  (in  Ideal  Time). 

•  Estimates  of  what  work  will  be  in  each  release. 

•  Sprint  Backlog. 

•  Project  Taskboa rd . 

SP  2.4 

Plan  for  necessary  resources  to 
perform  the  project. 

•  Scrum  estimates  in  Ideal  Time 

•  Release  Plan,  Sprint  Backlog  and  assignments. 

SP  2.6 

Plan  the  involvement  of  identified 
stakeholders . 

•  Scrum  process  roles  (including  team,  Scrum 

Master,  Product  Owner). 

•  [Note:  The  stakeholders  listed  in  Scrum  might  not 
be  the  complete  list  of  stakeholders  for  the  project, 
e.g.,  customers,  other  impacted  teams.] 

SP  3.2 

Reconcile  the  project  plan  to 
reflect  available  and  estimated 

resources. 

•  Sprint  Planning  meetin  g  . 

•  Daily  Scrum  meeting. 
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PMC 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Monitor  the  actual  values  of 
the  project  planning 
parameters  against  the 
project  plan. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining. 

•  Release  Burndown  chart  that  tracks  completed  story 
points.  This  shows  how  much  of  the  product 
functionality  is  left  to  complete. 

•  Project  Task  Board  used  to  track  stories 
(requirements)  that  are  done,  in  progress,  or  ones 
that  need  verification. 

SP  1.2 

Monitor  commitments  against 
those  identified  in  the  project 
plan. 

•  Discussions  on  team  commitments  at  the: 

-  Daily  Scrum  meeting. 

-  Sprint  Review  meeting. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining. 

•  Release  Burndown  chart  that  tracks  Story  Points  that 
have  been  completed.  This  shows  how  much  of  the 
product  functionality  is  left  to  complete. 

SP  1.6 

Periodically  review  the 
project's  progress, 
performance,  and  issues. 

•  Daily  Scrum  meetin  g . 

•  Sprint  Review  meeting. 

•  Retrospectives. 

SP  2.3 

Manage  corrective  actions  to 
closure. 

•  Tracking  of  actions  from: 

-  Daily  Scrum  meetin  g . 

-  Sprint  Review  meeting. 

•  [Note:  This  assumes  that  teams  will  track  (and  not 
lose)  actions.] 

No  risk  assessment  /  tracking  in  Scrum 

[SP  1 .3  Monitor  risks  against  those  identified  in  the  project  plan] 
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Burndown  Charts 


Release  Burndown  Chart 


Implements  PMC  spl.1 

Monitor  the  actual  values  of  the  project  planning  parameters  against  the  project  plan. 
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Measurement  and  Analysis 


SP  1.2 

Specify  measures  to 
address  the 
measurement 
objectives. 

•  Sprint  Burndown  chart  that  tracks  effort 
remaining. 

•  Release  Burndown  chart  that  tracks  story 
points  that  have  been  completed.  This  shows 
how  much  of  the  product  functionality  is  left  to 
complete. 

•  [Note:  These  two  measures  could  be  used  to 
track  the  progress  of  declared  project 
objectives,  such  as  “On  time,"  and  “On 
budget."] 

SP  1.4 

Specify  how 
measurement  data 
will  be  analyzed  and 
reported . 

•  The  Scrum  process  does  describe  the  purpose 
and  use  the  Sprint  and  Release  Burndown 
charts. 

•  [Note:  CMMI  expects  clearly  defined  analysis]. 

SP  2.1 

Obtain  specified 
measurement  data. 

•  Daily  Scrum  meeting  where  Sprint  and  Release 
Burndown  data  are  collected. 

SP  2.2 

Analyze  and  interpret 
measurement  data. 

•  Daily  Scrum  meeting  where  Sprint  and  Release 
Burndown  data  are  analyzed. 

SP  2.4 

Report  results  of 
measurement  and 
analysis  activities  to 
all  relevant 
stakeholders. 

•  Daily  Scrum  meeting  where  Sprint  and  Release 
Burndown  charts  are  reviewed. 

•  [Note:  Not  all  interested  stakeholders  will 
necessarily  be  at  the  Scrum  meeting.] 
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How  About  the  Other 
Components  of  Level  2? 

•  Configuration  Management  (CM): 

-  CM  is  not  specifically  called  out  in  Scrum.  However,  in  an  Agile 
environment  it  is  pretty  easy  to  add  a  layer  of  CM  to  protect  your 
work. 

•  Product  and  Process  Quality  Assurance  (PPQA): 

-  Some  basic  PPQA  activities  are  being  done  naturally  when  the 
Scrum  Master  checks  that  the  Scrum  process  is  being  followed. 

-  Scrum  does  not  specifically  call  out  a  level  of  objective  process  and 
product  check,  nor  does  it  state  that  particular  standards  or 
processes  should  be  defined  and  used. 

•  Supplier  Agreement  Management  (SAM): 

-  Not  included  in  Scrum. 
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Generic  Practices? 

•  Approximately  half  of  the  Level  2  GPs  of  REQM,  PP,  PMC  and  MA 
are  implemented  by  Scrum. 


GP  2.2 

Establish  and  maintain  the  plan 
for  performing  the 
REQM/PP/PMC/MA  process. 

•  The  Scrum  lifecycle  definition  and 
the  milestones  to  perform  Scrum. 

GP  2.3 

Provide  adequate  resources  for 
performing  the 

REQM/PP/PMC/MA  process, 
developing  the  work  products, 
and  providing  the  services  of  the 
process. 

•  The  resources  and  schedule  time 
allocated  to  perform  Scrum  planning, 
monitoring  and  requirements 
activities. 

GP  2.4 

Assign  responsibility  and  authority 
for  performing  the  process, 
developing  the  work  products, 
and  providing  the  services  of  the 
REQM/PP/PMC/MA  process. 

•  The  resource  assignments  allocated 
to  perform  Scrum  planning, 
monitoring  and  requirements 
activities. 

GP  2.6 

Place  designated  work  products 
of  the  REQM/PP/PMC/MA 
process  under  appropriate  levels 
of  control. 

•  [Note:  Scrum  does  not  explicitly 
require  CM  to  be  done.  However, 
this  can  be  performed  using  a  digital 
camera,  backed  up  drive,  or  share 
drive  with  versioning  and  controls 
turned  on.] 

GP  2.8 

Monitor  and  control  the 
REQM/PP/PMC/MA  process 
against  the  plan  for  performing 
the  process  and  take  appropriate 
corrective  action. 

•  Scrum  Master  monitoring  that  the 
steps  of  Scrum  are  followed. 
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How  About  Level  3? 

•  The  following  Level  3  components  are  not  readily  implemented  by 
Scrum  without  additional  work: 

-  Organizational  Process  Focus 

-  Organizational  Process  Definition 

-  Organizational  Training 

-  Integrated  Project  Management 

-  Risk  Management 

-  Decision  Analysis  and  Resolution 

-  Engineering  PAs  (e.g.,  RD,  TS,  PI,  VER,  VAL) 

-  Generic  Goal  3  (i.e.,  using  an  organization-wide  and  tailored 
process  with  measurements  and  lessons-learned) 
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Scrum  +  s 

+  2-4  week  cycles  creates  team  momentum,  and 
early  feedback  on  progress  and  technical 
solutions. 

+  Scrum  process  can  be  learned  and  used  in  less 
than  2  days. 

—  Speed  can  be  mistaken  for  progress: 

-  There  is  no  “Get  good  requirements”  phase,  only 
“Get  a  list  of  1 -liners  and  prioritize.”  (Although 
some  teams  do  more  than  that.) 

-  There  is  no  architecture  /  analysis  phase,  so  you 

could  implement  yourself  into  a  corner.  s\ 


-  This  is  fixable  by  making  the  focus  of  each  Sprint 
different. 

—  Applying  Scrum  to  large  teams  and  systems  takes 
extra  work. 

-  e.g.,  System  definition,  integration  and  coordination. 

©  Copyright  2004-2010  The  Process  Group.  All  rights  reserved.  www.processgroup.com 
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Summary 

*  Scrum  is  a  good  implementation  for  many  of  the 
practices  in  Level  2. 

*  A  group  can  easily  use  Scrum  and  CMMI  together. 

*  All  the  remaining  practices  in  Levels  2  and  3  can  be 
implemented  while  using  Scrum. 

*  An  organization  at  Level  2  or  3  could  adopt  Scrum  as 
an  additional  lifecycle  choice. 
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Agenda 

•  Introduction 

•  Applying  Requirements  Management  (REQM)  and  traceability 

•  Applying  Work  Planning  (WP)  and  Work  Monitoring  Control  (WMC)  to  the 
operation  of  a  services  group 

•  The  relation  between  Service  Delivery  and  WP  /  WMC 

•  Size  (attribute  estimation) 

•  Which  suppliers  to  apply  Supplier  Agreement  Management  (SAM)  to? 

•  What  is  being  audited  for  Process  and  Product  Quality  Assurance 
(PPQA)? 

•  How  to  apply  Configuration  Management  (CM)  to  service  artifacts 

•  Overlap  between  core  and  service  PAs  at  Level  2 

•  Appraisal  issues 

•  Reaction  of  a  new  group  to  the  SVC  model 

•  Suggestions  to  make  life  easier 

•  Summary 
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\ 

Just  apply  the 
GPs  to  the  PA 


What  is  SD 
GP2.2  vs.  WP 
GP2.2? 


J 


Presenting 


Appraising 


Presenting: 

-  Easy  to  gloss  over  all  the  sticky  issues  -  “Look  how  good  the  service  PAs  are.” 

Appraising: 

-  Sort  out  issues  such  as:  core  PA  interpretation,  overlap. 
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•  Appraisal  team: 

-  Three  people  experienced  with  DEV  (5-10  years)  +  LA  (22  years  with  DEV) 

-  Used  CMMI  1.2.  (This  presentation  uses  1.3  text  for  clarity.) 

•  Larger  group: 

-  200  people  that  design  and  build  large  airport  baggage  /  parcel  systems: 

»  Motors,  steel,  conveyors,  electronics,  software,  installation,  testing. 

•  Groups  appraised: 

-  Bids/Proposals  (7  people)  and  Financial  services  (11  people). 
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2  Managed 


Services  Model  vl  .3  -  Staged 


5  Optimizing 

4  Quantitatively 
Managed 

3  Defined 


Continuous 
Process 
Improvement 

Quantitative 
Management 

Process 
Standardization 


Basic 

Project 

Management 


Organizational  Performance  Management  (OPM) 
Causal  Analysis  and  Resolution  (CAR) 

Organizational  Process  Performance  (OPP) 
Quantitative  Work  Management  (QWM) 

Capacity  and  Availability  Management  (CAM)  (svc) 
Incident  Resolution  and  Prevention  (IRP)  (svc) 
Service  System  Transition  (SST)  (svc) 

Service  Continuity  (SCON)  (svc) 

Service  System  Development  (SSD)  (svc,  optional) 
Strategic  Service  Management  (STSM)  (svc) 
Organizational  Process  Focus  (OPF) 
Organizational  Process  Definition  (OPD) 
Organizational  Training  (OT) 

Integrated  Work  Management  (IPM) 

Risk  Management  (RSKM) 

Decision  Analysis  and  Resolution  (DAR) 

Service  Delivery  (SD)  (svc) 

Requirements  Management  (REQM) 

Work  Planning  (WP) 

Work  Monitoring  and  Control  (WMC) 

Supplier  Agreement  Management  (SAM) 
Measurement  and  Analysis  (MA) 

Process  and  Product  Quality  Assurance  (PPQA) 
Configuration  Management  (CM) 


Core  PAs 
SVC  PAs 


1  Initial 


Based  on  SEI  CMMI  Services  model 
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Applying  REQM  and  Traceability 


The  purpose  of  Requirements  Management  (REQM)  is  to  manage  requirements  of 
products  and  product  components  and  to  ensure  alignment  between  those 
requirements  and  the  work  plans  and  work  products. 


•  REQM  Takes  A  LOT  of  explaining  to  a  non-familiar  services  group: 

-  The  services  groups  were  bidding/tracking  product  requirements. 

»  Caused  total  confusion  when  they  were  left  by  themselves  to  read  CMMI. 

-  Requirements  defined  as  group  roles  &  responsibilities. 

-  Initially  we  had  to  explain  why  REQM  and  SD  are  separate. 

»  Then  we  mapped  REQM  and  SD  together  -  provide  one  front  to  the  appraisal  team 
and  organization. 

-  Too  much  GP!:  Policy,  plan,  training,  monitoring  and  auditing  of  roles  & 
responsibilities  definition  activities? 

»  REQM  might  only  take  1  day  per  year  and  1  update  every  6  months. 

»  The  GPs  need  to  be  scaled  down  DRAMATICALLY  to  be  useful  OR  mapped  to  SD 
GPs.  We  merged  REQM  and  SD  GPs. 
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Explaining  REQM  in  the  Appraisal 


•  The  appraisal  team  re-wrote  the  PA  purpose  for  the  final  findings 
presentation: 

-  The  purpose  of  Requirements  Management  (REQM)  is  to  manage 
requirements  of  products  and  product  components  and  to  ensure 
alignment  between  those  requirements  and  the  work  plans  and 
work  products. 

-  The  purpose  of  Requirements  Management  (REQM)  is  to,  a)  define 
the  services  of  the  group,  b)  trace  defined  services  to  team 
activities,  c)  verify  that  resources,  service  definition  and  actual  work 
done  are  aligned,  [appraisal  team  definition] 
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Bidirectional  Traceability  (spi.4) 

SP  1 .4  Maintain  bidirectional  traceability  among  requirements  and  work  products. 

“In  a  service  environment,  you  should  be  able  to  trace  stakeholder  requirements  to  the 
elements  of  the  delivered  service  and  supporting  service  system  that  were  developed  from 
those  requirements ”  [cmmi  1.2  &  1. 3] 

Example  1 


SOX  Requirement 

Implementation 

Test 

SOX  Annex  -  section  1 

Finance  role  1,  report  1 

SOX  test  1 

SOX  Annex  -  section  2 

Finance  role  2,  report  2 

SOX  test  2 

SOX  Annex  -  section  3 

Finance  role  3,  report  3 

SOX  test  3 

Example  2 


Bid  Role 
(defined  in  SLA) 

Authority  Level 

Tasks  for  Role 

Role  1 

Approve  up  to  $X 

Obtain  estimates,  define  bid,  check  bid 

Role  2 

Approve  up  to  $Y 

Tasks  -  role  2 

Role  3 

Approve  up  to  $Z 

Tasks  -  role  3 
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Applying  WP  and  WMC 

•  The  group  performs  annual  resource  planning. 

•  “The  plan”  =  Annual  resource  plan  and  service-event  plan. 

•  Risks  are  assessed  monthly: 

-  E.g.,  Do  we  have  resources  to  cover  each  bid  /  financial 
report? 

•  Schedules  consist  of  bid  and  financial  report  deadlines. 

•  Stakeholders  are  defined  on  approval  and  signature  sheets. 

•  Too  much  GP!:  policy,  plan,  training,  monitoring  and 
auditing  of  annual  and  service-event  planning? 

-  Most  planning  events  were  a  few  hours  in  length. 

-  We  merged  WP  and  WMC  GPs. 

-  Audits  =  signature  approvals  with  checklists. 
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Relation  Between  SD  and  WP  /  WMC 


•  In  SD,  planning  (GP2.2)  is  used  to  plan  the  readiness  and 
operation  of  a  services  group: 

»  SD  GP2.2  -  Establish  and  maintain  the  plan  for  performing  the  process. 

»  SD  GP2.3  -  Provide  adequate  resources  for  performing  the  process . 

»  SD  GP2.4  -  Assign  responsibility  and  authority  for  performing  the  process . 

»  SD  GP2.8  -  Monitor  and  control  the  process  against  the  plan  for  performing  the  process 
and  take  appropriate  corrective  action. 

•  So  what  was  WP  /  WMC  used  for? 

-  Operations  planning,  of  which  service  delivery  is  one  significant  aspect. 

-  Annual  resource  planning  and  monthly  resource  tracking. 

-  Special  projects  (non-trivial  “other”  work). 
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Size  (Attribute  Estimation) 


•  The  group  reads  the  practice: 

-  WP  SP  1 .3  Establish  and  maintain  estimates 
of  work  product  and  task  attributes. 

-  And  assumes  that  it  is  #Feet  of  steel, 

#motors . etc. 

•  The  bid  group  estimates  the  cost  of  a 
project: 

-  #Feet  of  steel,  #motors,  #control  panels, 
installation  labor. 


•  Luckily: 

-  Finance  already  had: 

»  Project  categorization  (the  size/complexity  of  the  project  being  financially  tracked). 

-  Bids  already  had: 

»  Bid  volume  (#bid  requests  likely  to  arrive  per  month). 
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Which  Suppliers  to  Apply  SAM  to? 

“The  scope  of  this  process  area  addresses  the  acquisition  of  products,  services, 
and  product  and  service  components  that  can  be  delivered  to  the  service's 
customer  or  included  in  a  product  or  service  system.  This  process  area’ s 
practices  can  also  be  used  for  other  purposes  that  benefit  the  service  (e.g., 
purchasing  consumables).”  [CMMI  1.3] 


•  Both  services  teams  had  no  suppliers: 

-  An  example  would  have  been  accounting  experts, 
proposal  writers. 

•  The  product  team  had  hundreds  of  suppliers;  this 
was  appraised  under  SAM  in  the  DEV  model. 
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What  is  Being  Audited  for  PPQA? 


•  Processes: 

-  Lifecycle  and  Service  Delivery  processes. 

-  Process  Area  processes. 

•  Work  products: 

-  Service  deliverables  +  critical  internal 
documents. 


Processes  audited 
via: 

•  Extensive 
Management 
Approvals. 

•  SOX  audits. 

•  Corporate  finance 
audits. 


Request  ,  | 
forbid  ^ 


Bid 

process 


Bid  award 


Financial 

tracking 

process 


£ 

Bid 


Company  Lifecycle 


& 

Monthly  $ 
reports 


ISO  audits. 
Random  quizzes. 


Documents  audited 
via: 

•  Peer  reviews  using 
checklists. 


•  Signature  approvals 
of  document  content. 
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How  to  Apply  CM  to  Service  Artifacts 


•  Identify  documents  that  service  groups  care  about,  e.g.,: 

-  Internal:  Annual  plan,  checklists,  service  agreement,  audit  results. 

-  Deliverable:  Requests,  proposals,  estimate  sheets,  contracts. 

•  Define  directory  structure  for  all  documents,  and  plan  for  archival. 

-  Merge  Data  Management  plan  in  with  CM. 

•  GPs  -  keep  them  simple! 

-  Someone  assigned  to  set  up  the  folders  and  access. 

-  Task  defined  to  “establish  CM”  that  can  be  planned/tracked. 

-  Team  meetings  for  training,  random  audits  for  objective  evaluation. 


Bid 

process 


Financial 

tracking 


Bid  award  process 


& 


Bid 


Monthly  $ 
reports 
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Overlap  Between  Core  and  Service  PAs 

*  Lots  of  overlap  -  a  MUST  fix  before  appraising,  otherwise: 

-  You  will  be  asking  the  same  question  3-4  times. 

-  The  appraisal  team  will  wear  out. 

-  The  interviewees  will  think  you  are  nuts  and  unsure  about  the  model. 


Practice  # 

REQM  Practice  Definition 

Rewrite  (Italics  =  exact  copy  of  CMMI  practice) 

r  H 

Develop  an  understanding  with  the 
requirements  providers  on  the  meaning 
of  the  requirements. 

See  SD  spl  .2 

7  12 

Obtain  commitment  to  the  requirements 
from  participants. 

See  SD  spl  .2 

r  1.3 

Manage  changes  to  the  requirements 
as  they  evolve. 

See  SD  spl  .2 

7  Ta 

Maintain  bidirectional  traceability  among 
requirements  and  work  products. 

Trace  service  requirements  to  downstream 
group  activities  and  back  (e.g.,  map  service 
descriptions  to  team  activities  and  deliverables; 
do  we  implement  our  service  definition,  do  we 
exceed  it,  do  we  have  gaps?) 

r  1.5 

Ensure  that  plans  and  work  products 
remain  aligned  with  requirements. 

Align  service  definition,  schedule/budget,  team 
activities. 

Examples: 

*  Does  the  services  list  state  to  do  A,  but  in 
reality,  the  team  is  doing  B. 

‘Analyze  measurement,  resource  and  service 
request  data. 
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Merging  Some  REQM  Practices  with  SD 


Practice  # 

SD  Practice  Definition 

Rewrite  (with  additions  from  REQM  SPs  and 
GPs.  Italics  =  exact  copy  of  CM  Ml  practice) 

1.1 

Analyze  existing  service  agreements 
and  service  data  to  prepare  for 
expected  new  agreements. 

Analyze  existing  service  agreements  and 
service  data.  Use  data  as  a  basis  for  new 
agreements.  What  can  we  achieve  based  on  the 
past  -  lessons;  overlooked  tasks?) 

(e.g . ,  In  the  past,  what  has  been  the  sales 
volume,  #bids  that  can  be  processed, 
timeliness.) 

1.2 

Establish  and  maintain  the  service 
agreement. 

Define  the  services  of  the  group  and  review  with 
staff.  [REQM  sp  1.1], 

Obtain  commitment  to  the  services  definition 
from  the  staff.  [REQM  sp  1.2] 

Establish  and  maintain  the  service  agreements 
with  customers. 

Manage  changes  to  services  (evaluate  change). 

©  Copyright  2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


vl.Ob 


16 


THE 

PROCESS 

GROUP 


Overlap  (Continued) 

REQM  (GPs) 


GP  2.1 

Establish  and  maintain  an 
organizational  policy  for  planning  and 
performing  the  process. 

See  SD  GP  2.1 

GP  2.2 

Establish  and  maintain  the  plan  for 
performing  the  process. 

See  SD  GP  2.2 

GP  2.3 

Provide  adequate  resources  for 
performing  the  process,  developing  the 
work  products,  and  providing  the 
services  of  the  process. 

See  SD  GP  2.3 

GP  2.4 

Assign  responsibility  and  authority  for 
performing  the  process,  developing  the 
work  products,  and  providing  the 
services  of  the  process. 

See  SD  GP  2.4 

Practice  # 

WP  Practice  Definition 

Rewrite  (with  additions  from  WMC  GPs.  Italics  = 
exact  copy  of  CMMI  practice) 

1.1 

Establish  and  maintain  the  service 
strategy. 

Objectives:  See  MA  spl  .1 . 

Approach:  See  SD  sp  2.1. 

Risks:  See  WP  sp  2.2. 
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Appraisal  Issues 

•  Asking  CMMI  questions  to  service  professionals  unfamiliar  with 
CMMI: 

-  Plan  on  rewriting  (some  of)  the  CORE  PA  practices,  so  that  the 
interpretation  is  consistent  and  uses  easier  words. 


Practice  # 

OM  Practice  Definition 

Rewrite  (Italics  =  exact  copy  of  CMMI  practice) 

1 . 1 

Identify  the  configuration  items, 
components,  and  related  work  products 
to  be  placed  under  configuration 
management. 

Define  a  list  of  items  (files,  contracts,  bids, 
deliverables,  documents)  for  the  group  that  will 
be  placed  under  control  (e  g.,  version,  access, 
backup). 

1.2 

Establish  and  maintain  a  configuration 
management  and  change  management 
system  for  controlling  work  products. 

Establish  and  maintain  a  system  (manual  or 
automated)  to  store,  manage  and  protect  files 
on  the  list  above. 

1.3 

Oreate  or  release  baselines  for  internal 
use  and  for  delivery  to  the  customer. 

Define  naming  conventions  for  files  and 
documents,  as  they  are  created  and  released. 

2. 1 

Track  change  requests  for  the 
configuration  items. 

Track  changes  made  to  files  under  OM  control. 

Exam  pie: 

*  Track  changes  in  a  history  file  or  comment 
field  of  a  version  tool. 

2.2 

Oontrol  changes  to  the  configuration 
items. 

Establish  read/write  access  for  files  under  OM 
control . 

•  Some  service-specific  practices  need  examples  too: 

-  e.g.,  “Maintain  the  service  system”  means? 
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Reaction  of  New  Group  to  the  SVC  Model 

Target  audience  reaction: 

-  “How  does  this  relate  to  our  function?” 

»  “What  are  work  product  attributes,”  “What  risks?,”  What  are 

configuration  items?”  “What  is  a  lifecycle?” 

»  Lead  appraiser  had  to  advise  “don’ t  read  the  model,  it  will  just  confuse 
you  more.”  [specifically  the  core  PAs] 

Appraisal  team  reaction  : 

-  Initially,  total  confusion: 

»  Reading  the  model,  and  memorizing  what  each  practice  means  was 
wearing. 

»  “Model  needs  redoing,”  -  wording  and  overlap  of  practices. 

»  The  findings  of  the  class  B  appraisal  acted  as  the  new  model  definition 

for  the  team,  along  with  the  lead  appraisers  model  rewrite. 


The  core  PAs  are  the  challenge 
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Suggestions  to  Make  Life  Easier 


*  Run  a  (very)  informal  appraisal  first,  so  that: 

-  Interviewees  have  some  idea  of  what  your  model  interpretation  is. 

-  You  can  obtain  experience  asking  interview  questions  and 
understanding  the  responses. 


•  Clarify  terms  (model  glossary  might  /  might  not  help*): 

-  “Requirement,”  “stakeholders  in  planning,”  “risk”  vs.  “issue,” 
“traceability,”  “configuration  item”  “monitor  the  monitoring  process.” 


•  Train  your  team  in  the  interpretation  of  the  model  before  you  appraise: 


-  The  SVC  Supplement  class  doesn't  cover  the  core  PAs. 


-  The  Intro  to  CMMI  SVC  class  doesn’t  cover  overlap  between  PAs  and 
the  GPs  in  detail. 


*Requirement:  (1 )  A  condition  or  capability  needed  by  an  end  user  to  solve  a  problem  or 
achieve  an  objective.  (2)  A  condition  or  capability  that  must  be  met  or  possessed  by  a  product, 
service,  product  component,  or  service  component  to  satisfy  a  supplier  agreement,  standard, 
specification,  or  other  formally  imposed  documents.  (3)  A  documented  representation  of  a 
condition  or  capability  as  in  (1)  or  (2). 
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Summary 

•  The  core  Maturity  2  Level  PAs  CAN  be  used  in  a  services 
organization.  They  do  work. 

-  An  organization  at  ML2  is  a  more  organized  and  efficient: 

»  E.g.,  #Mistakes,  Response  time,  #Hours  expended. 

•  The  core  PA  practices  are  written  for  development,  not  for 
services. 

-  Intro  &  Supplement  classes  focus  on  the  benefit  of  the  service  PAs, 
not  the  difficulty  interpreting  the  core  PAs. 

-  Plan  on  a  rewrite,  otherwise: 

»  a)  You  will  have  to  explain  practices  every  time. 

»  b)  The  target  audience  will  probably  forget  the  meaning  and  get  totally  off  track. 

•  Define  what  services  or  work  the  PAs  are  being  applied  to. 


©  Copyright  2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


vl.Ob 


21 


THE 

PROCESS 

GROUP 


Questions? 
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•  SOX  -  Sarbanes  Oxley 

*  SLA  -  Service  Level  Agreement 

•  WP  -  Work  Planning 

*  WMP  -  Work  Monitoring  and  Control 


©  Copyright  2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


vl.Ob 


23 


Raytheon 


CMMI 


11™  ANNUAL 

CMMI®  TECHNOLOGY 
CONFERENCE  AND 
USER  GROUP 


Some  Assembly  Required:  Using 
Agile  Methodologies  to  Develop  an 
Interactive  Software  User’s  Guide 


Presentation  Agenda 

■  Some  Assembly  Required:  Agile  Methodologies 


Raytheon 


■  Introduction  /  Problem  Statement 

-  Why  pursue  a  new  technical  document  development  platform? 

■  Part  1...  Background:  Enabling  Technologies,  Software 
Architecture  and  Development  Platforms 

-  Object-oriented  programming  (OOP) 

-  Rich  Internet  Application  (RIA)  /  Rich  Desktop  Application  (RDA) 

■  Part  2...  SUG  Development 

-  Agile  and  Technical  Documentation 

-  The  Pilot  Project 

-  User  stories 

■  Part  3...  SUG  Lessons  Learned 
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Why  Pursue  a  New  Development  Platform? 


■  Current  development  platform  (Authorware)  is  ‘abandonware’ 

■  Vendor  (Adobe)  discontinued  development  in  August  of  2007 

■  Strengths 

■  broad  acceptance  in  the  marketplace;  rapid  prototyping; 
particularly  well  suited  to  creating  e-learning  content 

■  Weaknesses 

■  XML  handling  is  sub-optimal;  web  delivery  and  desktop  delivery  is 
problematic;  ‘Protection  Mode’  and  runtime  errors  with  IE7  and  Vista 


■  Need  a  development  platform  that’s  ‘future-proofed’ 

■  leverage  Broadband  proliferation  and  enabling  technologies 

■  accommodate  multi-channel  delivery: 

web,  desktop,  mobile,  and  print  ■*:  p 
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■  What  is  an  Object  Oriented 


A  computer  language  can  be  said  to  be  Object  Oriented  if  it  provides  support  for  the  following: 

Class 

A  class  is  a  blueprint,  or  prototype,  that  defines  the  variables  and  the  methods 
common  to  all  objects  of  a  certain  kind. 

Object 

An  instance  of  a  class.  More  than  one  instance  of  the  same  class  can  be  in 
existence  at  any  one  time. 

Encapsulation 

The  act  of  placing  data  and  the  operations  that  perform  on  that  data  in  the  same 
class.  The  class  then  becomes  the  container  for  the  data  and  operations. 

Inheritance 

The  reuse  of  base  classes  to  form  derived  classes  (subclasses).  Methods  and 
properties  defined  in  the  superclass  are  automatically  shared  by  any  subclass. 

Polymorphism 

ability  of  objects  belonging  to  different  types  to  respond  to  method,  field,  or  property 
calls  of  the  same  name,  each  one  according  to  an  appropriate  type-specific  behavior. 

■  Why  use  it? 

■  OOP  is  an  efficient  methodology  for  minimizing 
development  time  and  maximizing  code  reusability. 

■  OOP  creates  a  modular  design  that  is  easily  modified 
without  having  to  restructure  the  entire  system. 


Programming  language? 

Programming  oriented  around  objects  -  self-contained  collections 
of  computational  procedures  and  data  structures. 
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Rich  Internet  Application  (RIA) 


Web-based  application  that  approaches  the  speed 
and  elegance  of  a  local  (installed)  application 

■  Traditional  desktop  features 

■  Drag  and  drop,  windows, 
wizards,  and  panels 


i 

3. 

UHR 

CfpLf-  *  _ 

JTM  %  DM 

uMM 

•1  B  ”  J 

o  BaM 

|  PH*-  I  IVhI  I  rrJ  I 

Cross  Platform 

n  Cross  Browser 

n  IE,  Mozilla  Firefox,  Safari 
■  Cross  Operating  System 

n  Windows,  Linux,  MAC  OS 


No  installation,  and  accessible  just  about  anywhere 


MVC  pattern  on  client  and  server 

n  MVC  pattern(s)  can  manage  the  interaction  between  user  and  the  user  interface, 
and  requests  to  the  server 
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Rich  Desktop  Application  (RDA) 

Platform-independent  web  applications 
that  can  be  run  on  a  user's  desktop 


In  contrast  to  RIA’s,  RDA’s  are  able  to  run  off-browser  and  off-line 


Applications  have  network 
connection  awareness 

Cross  Platform...  browsers 
and  operating  systems 


Desktop  security  model  -  applications 
can  be  packaged  and  digitally  signed 


*5  Application  Install 


EM® 


■H  Are  you  sure  you  want  to  install  this 

UKi  application  to  your  computer? 

Ht3']  m  i  '/ji  \ 


Publi-sher:  Raytheon 
Application:  Radar  SUG 
astern  Access:  UNRESTRICTED 


Publisher  Identity:  VERIFIED 
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Feature 

RIA  in  the  Browser 

RDA  on  the  Desktop 

Application  delivery 

Applications  can  be  easily  discovered, 
explored,  and  used. 

Installed  applications  have  more  persistence,  power, 
and  functionality. 

Installation 

No  application  installation  is  necessary. 

Applications  install  seamlessly  from  the  browser  or 
install  like  a  traditional  application 

Background 

capability 

RIAs  can  only  run  in  a  visible  browser 
window 

Applications  can  run  in  the  background  and  provide 
notifications  like  traditional  desktop  applications 

Persistence 

Activity  is  limited  to  the  browser  session. 
When  the  browser  is  closed,  information  is 
lost. 

RDAs  are  installed  on  the  desktop.  They  store 
information  locally  and  operate  offline. 

Desktop  integration 

Applications  are  sandboxed,  so  desktop 
integration  is  limited. 

Applications  can  access  a  desktop  file  system, 
clipboard,  drag  and  drop  events,  system  notifications, 
and  more. 

User  interface  control 

RIAs  run  within  a  browser  window  that  has 
its  on  controls,  branding,  and  integration 
with  the  desktop. 

RDAs  have  a  customizable  user  interface  and 
desktop  integration,  enabling  branding  experiences. 

Data  storage 

Applications  have  limited  local  storage, 
which  the  browser  can  destroy. 

Applications  have  unlimited  local  storage  and  access 
to  a  local  database,  plus  encrypted  local  storage 
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■  Adobe  Flash/Flex/AIR 


< 

AOOBE  AIR 


■  Sun  JavaFX 

■  Microsoft  Silverlight 

■  Google  GWT 

■  OpenLazzIo 

■  AJAX 


t  Javafx 


0 


i- 

Silvertight 


OpenLaszIo 


atTax 

kWXhrw&j#  MvM&pt  And 


"By  2010,  at  least  GO  percent  of  new  application  development 
projects  will  include  RtA  technology,  and  at  least  25 
percent  of  those  wilt  rely  primarily  on  RIA,“ 

Source:  Gartner  RAS  Core  Research 
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■  Flash  Builder 


■  Adobe’s  platform  for  developing  and  deploying 
RIAs  for  the  browser 

■  Leverages  the  nearly  ubiquitous  Flash  Player 
(98%  penetration  level) 

■  Flex  applications  can  be  built  using  the  free 
open  source  Eclipse  framework 


Adobe  Fie  x  3  Comp  sue  nl  Explorer 

t  Q Visual  Component 

►  _]  Gcneisl  ConCrgls 
p- 12 1  BuTffin  Cpntror^ 

► 'jaQaW  CanEml^ 

*  fj  L  Odder  C*ntT&l£ 

►  Ms, iiu  Controls 

*  Qj  lejtt:  Controls 

►  Conta^nejs 

►  Qj  Repeater  Control 
f  print  Controls 

|_j  Flex  Pri  nUob,  Rri  n XDaia  Qrv 
j  *  Validate  re  and  fo matters 


Adobe  AIR  (Adobe  Integrated  Runtime) 

■  Adobe’s  platform  for  deploying  RIAs  for  the  desktop 

■  Allows  you  to  transform  a  RIA  into 
a  Rich  Desktop  Application  (RDA) 

■  Air  applications  run  outside  of  the 
browser,  on  multiple  operating 
systems 


^  O  Va  Hd  sto  rs 
»n  Formatters 
t  iisrfcfCfrtft,  vitwStawf,  and  tra! 
►  u  effects- 
^  £3  View  Stites 
t  ''  a  TMnsio*ns 


mp&nen 
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Flash  Builder  and  Adobe  Air...  Who’s  Using  It? 


Application 


■  Flash  Builder  and  Adobe  AIR  are  reaching  critical  mass 


■  SAP,  HP,  Google,  NASDAQ,  eBay,  AOL,  Yahoo,  MINI... 
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Agile  and  Technical  Documentation 
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■  Traditionally,  Agile  methodologies  focused  on  software 
development  or  product  engineering 

■  Recently,  Agile  practices  have  extended  to  other  areas  such  as 
technical  documentation 

■  Agile  methodologies  encourage/allow: 

■  A  means  to  accommodate  content  that  is  more  and  more  malleable  and 
requires  more  touch  points 

■  Effective  communication  and  collaborative  spaces  for  technical 
communicators  and  Subject  Matter  Experts  (SMEs) 

■  The  use  of  Scrums  and  user  stories  to  create  a  platform  by  which  to 
develop  customer-facing  documentation 
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Pu  ration 


Pi cJfz  this 

project 


Short 


Small 
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■  Duration 

■  A  project  that  could  be  completed  in  8  weeks 

■  Project  Size 

■  A  project  that  could  be  supported  by  one  team 


■  Importance 

■  Medium  priority 

■  Business  Sponsor  Engagement 

■  Reasonably  strong 
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Agile  and  User  Stories 

■  Substitute  of  formal  requirements  documents 

■  Summaries  of  functionality  that  leave  space  for  expansion  and 
refinement 

■  Prioritized  based  on  costs  and  their  value 

As  a  designer,  I  can  customize  the  look  and  feel  of  my  deliverable 

As  a  user,  I  can  drill  down  to  specific  information  using  multi-level 
menus 

As  a  user,  I  can  display/use  the  SUG  on  multiple  platforms 
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Software  User’s  Guide  (SUG) ...  Prototype 


Skinning  and  styling 

■  As  a  designer,  I  can  customize  the 
look  and  feel  of  my  deliverable 


RTCP &  DCP 


Software  User’s  Manual  -  Version  1.3 


Acronym 


Fx»  Flex  Style  Explorer  v  3.0  BETA 


Background  Image 
Background  Color 
Background  Gradient  Colors 
Background  Gradient  Alphas 
Theme  Color 
Text  Color 


brushedmetal 


m 


Restore  Defaults 
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Application  -[ 

backgro  undlmage: 

Embed(sou  rce  = 11  a  s  s  ets/  b  ru  s  h  e  d  i 

ipg"3f 

backgroundSi 
backgroundCi 
backgroundGi 
#cc3333; 

themeColor: 
color:  #ffffff; 


Introduction  > 

RTCP  &  DCP  Screens/Menus  ► 
CCLS  Tasks 

RTCP  Tasks  ► 


Button  -[ 

paddingRight:  6; 
letterSpacing:  lj 
highlightAlphas :  0.42,  O.lSj 
fillColors :  #000000,  #000000, 
#000000,  #eeeeeej 
color:  #ffffff; 
themeColor:  #005dffj 

} 


RTCP  &  DCP  Software  User's  Manual  -  Version  1.3 


The  RTCP  (RadarTest  Control  Program)  &DCP  (Display  Control  Program)  SUM 
is  a  menu  driven  technical  manual,  which  uses  pre-programmed  links  to  assist 
the  manual. 


Fa 1 1 It q /Pit At i  El  H  hafts? 


DCP  Alerts 
GLOBAL  Alerts 


any  1 1  SO  Hartwell  Road  |  Bedford,  Massachusetts 


RTCP& 


Software  User's  Manual  -  Version  1.3 


paddingRight:  6; 
letterSpacing:  lj 
highlightAlphas:  0,42,  O.lSj 
fillColors:  #000000,  #000000, 
#000000,  #eeeeee; 
color:  #ffffffj 
themeColor:  #009dff; 


RTCP  &  DCP  Software  User's  Manual  -  Version  1.3 


The  RTCP  (RadarTest  Control  Program)  &DCP  (Display  Control  Program)  SUM  (Software  User' 
is  a  menu  driven  technical  manual,  which  uses  pre-programmed  links  to  assist  the  operator  in  r 
the  manual. 


3ny  1 1  SO  Hartwell  Road  |  Bedford,  Massachusetts  01 730 
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Software  User’s  Guide  (SUG) ...  Prototype 
■  Versatile  Menu  System 

■  As  a  user,  I  can  drill  down  to  specific  information  using  multi-level  menus 

■  As  a  designer,  I  can  pull  data  from  external  XML  collections 


Introduction 

RTCP  Si  DCP  Screens/Menus 
CCLS  Tasks 


RTCP  Tasks 


Alerts 

Faults/Status  Charts 
System  Administration 
PDF  Library 


XML  Tret  Menu...  ActlonScrlpt 


RTCP  &  DCP  Softwaite  Usfir1*  f 


Th  e  RTCP  (R  a  d  a  r  Te  st  WEJ  Frog-MUFfl  ft 

is  a  menu  driven  tRnhrrfjBWMMHMMI 


Daily  Diagnostics 


-my  |  ISO  m 


Radar  System  Initializatior 
Satellite  List  Generation 
Satellite  Tracking 


gurdtmi 


xml.  on  Load  -  function^  [ 
theT  ree.  data  Provider  =  this.firslChild; 

}; 

xml.  load  ftree.  xml"); 


var  ireeL: Object  -  new  Objecl(); 


treeL.change  -  funclion()  { 

^ar  item  -  th  eTree.se  lected  I  tern 

van  earl  =  ilam.atlritJutBa.il rl: 


var  xml:  XML  -  newXMLQ; 
xml.  ignore  White  =  true, 


ar 

er 
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Software  User’s  Guide  (SUG) ...  Prototype 
■  Data  Grids  and  Tables 

■  As  a  user  I  can  resize,  and  sort  column  layouts 


RTCP 


<?xml  version=rrl.  0rr  encoding=rrUTF-8rr?> 

<dataroot> 

<Alert> 

<E  r  r  o  r_I  d_E  vent_I  d>D  cp_0p  s  c  ap_Up  date<  /E  r  r  o  r_I  d_E  vent_I  d> 
<Message_Text>OPSCAP  has  changed  &lt.;Neu_Status_St.ring£:gt.;</Message_Text> 
<Us e r_He lp_D at> Che ck  OPS C AP< /Use r_He lp_D at> 

•</Alert> 


PCP  |  G  l,  DEALS  |  MAP  I  DS*  |  FSS  j  KCR 


EiTorJd_Euent_ld 

Message_Text 

Action_Re(|uired 

User_Help_Dat 

Tr_Module_Command_Failure 

Faulty  antenna  Elements  not 
disabled 

undefined 

JR  Module  Cmd  processing  returned  status  of  & 

T utl_T est_Status  ,_T ypes  .T est_Results_Etype'lmage  (T r_Mc 

Timer_Expired_T  con 

The  transition  to  RF  disabled  has 
timed  out 

undefined 

<Delete  redundant  supplemental  data> 

Timer  Excired  Tcon 

The  transition  to  RF  enabled  has 

undefined 

<Delete  redundant  sucolemental  data> 
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Software  User’s  Guide  (SUG) ...  Prototype 
■  Multi-channel  delivery 

■  As  a  user,  I  can  display/use  the  SUG  on  multiple  platforms 
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Lessons  Learning 

Software  User’s  Guide  (SUG) ...  Prototype 
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■  Team 

■  Team  was  not  co-located 

■  Could  have  been  better  formed 

■  Short  stories 

■  Need  to  be  short 

■  Need  to  have  clear  value  to  the  user 

■  Divide  the  feature  (story)  into  smaller  pieces  and  add  value  incrementally 


■  Momentum 

■  When  the  team  sees  visible  progress,  progress  and  creativity  increase 

■  Relates  back  to  keeping  “stories  short”  and  fostering  team  communication 
and  demanding  whole  team  collaboration 
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Software  User’s  Guide  (SUG) ...  Prototype 
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■  Feedback 

■  Rapid  feedback  cycles  make  for  quicker  adjustments  and  improvement 

■  Easy  to  fall  into  a  vacuum  -  little  feedback  to  sanity  check  decisions 

■  Relates  back  to  keeping  “stories  short” 

■  Acceptance  testing 

■  In  hindsight  -  ad  hoc 

■  Different  hardware/software  platforms  led  to  “faux” 
behavior/performance/scenario  testing 

■  Conclusions 

■  Practice  iterative,  incremental  development 

■  Encourage  active  customer  participation  and  demand  whole  team 
collaboration 

■  Deliver  business  value  at  regular  intervals  at  a  sustainable  pace 
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Questions? 

Some  Assembly  Required:  Agile  Methodologies 
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|  Ron  Stauffer 

Senior  Tech  Stipori  Engineer  i 


401  Jan  Davis  Drive 
Huntsville,  AL  35758 
256.217  6376  Office 


256  797  2644  cell 


JhU  n-J  If  trJhn  * 

RTCP  &  DCP  S^fMfiWft^Tiirt  ► 

cos  Tal**  F 


RTCP  a  DCP  software  User'-*  t 

jrr 

Iht  HTCP  (Rid*  T*!l 


AJ&tl* 

FiiriLUStihn  CtitrH 
POF  LAf*/ 

Agile  Methodologies 


I  PjiPji 


I  Diagnhfiltfi 


;  .  ■ !  ■:  n  i  1 . 1 


P*Jl 

S^.'illJa  Lilt  Gcnurjlkin 
SatdU‘1*  TiiCt'r-p 


Radar 


ir*f  '■  | 

iMT'ii  cjEiPrai^lnr  ■* 

l 


d-ii  i^r.  AT.1.  i  b»yf. 
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Why  do  developers  make  dangerous  software  errors? 


Michele  Moss 
NDIA  CMMI®  Technology  Conference 

November  15,  2011 


Key  questions 


►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


Requirements  for  secure  code  are  implicitly  and  not  explicitly 
stated 

What  CIOs  want 


Reliabile  software  that  functions  as 
promised 

Software  free  from  security 
vulnerabilities  and  malicious  code 

Ease  of  Integration  &  Configuration 

Software  conforms  to  Requirements 
&  Industry  Standards 

Convenience  &  Ease  of  Use 


Rich  Feature  Set 


CIO  Executive  Council 

The  Professional  Organization  for  CIOs 


Other 


httos://www.  cjoexecutiyecguncil.  com  October  1 1,  2006  Press  Release 


□  Percent 


“Defacto”  security  requirements  in  NIST  800-53  do  not  explicitly 
require  developers  to  produce  secure  code 


►  Technical 

-  AC-2  Account  Management 

-  AC-3  Access  Enforcement 

-  AC-4  Information  Flow 
Enforcement 


►  Operational 

-  CM-7  Least  Functionality 

-  SI-3  Malicious  Code  Protection 

-  SI-10  Information  Input  Validation 


►  Management 

-  RA-5  Vulnerability  Scanning 


Technical  controls  in  NIST  800-53  contribute  to  application  security 


-  AC-7  Unsuccessful  Login 
Attempts 

-  AC-10  Concurrent  Session 
Control 

-  AC-1 1  Session  Lock 

-  AC-14  Permitted  Actions 
without  Identification  or 
Authentication 

-  AC  - 16  Security  Attributes 

-  AC-1 7  Remote  Access 

-  AC-20  Use  of  External 
Information  Systems 

-  AU-2  Auditable  Events 

-  AU-3  Content  of  Audit  Records 

-  AU-4  Audit  Storage  Capacity 

-  AU-9  Protection  of  Audit 
Information 

-  IA-3  Device  Identification  and 
Authentication 


-  IA-4  Identifier  Management 

-  IA-5  Authenticator 
Management 

-  IA-6  Authenticator  Feedback 

-  IA-7  Cryptographic  Module 
Authentication 

-  IA  -  8  Identification  and 
Authentication  (Non- 
Organizational  Users) 

-  SC-2  Application  Partitioning 

-  SC-3  Security  Function 
Isolation 

-  SC-4  Information  in  Shared 
Resources 

-  SC-7  Boundary  Protection 

-  SC-8  Transmission  Integrity 

-  SC-10  Network  Disconnect 

-  SC-11  Trusted  Path 


-  SC-12  Cryptographic  Key 
Establishment  and 
Management 

-  SC-24  Fail  in  Known  State 

-  SC-25  Thin  Nodes 

-  AC-1 8  Wireless  Access 

-  AC-1 9  Access  Control  for 
Mobile  Devices 

-  SC-9  Transmission 
Confidentiality 

-  SC-13  Use  of  Cryptography 

-  SC-28  Protection  of 
Information  at  Rest 

-  SC-23  Session  Authenticity 

-  AC-5  Separation  of  Duties 

-  AC-6  Least  Privilege 


Operational  controls  in  NIST  800-53  contribute  to  application 
security 


-  SI-1  System  and  Information 
Integrity  Policy  and  Procedures 

-  SI-2  Flaw  Remediation 

-  AT-2  Security  Awareness 

-  AT-3  Security  Training 

-  CM-3  Configuration  Change 
Control 

-  CM-4  Security  Impact  Analysis 

-  CM-5  Access  Restrictions  for 
Change 

-  SI-4  Information  System 
Monitoring 

-  SI-6  Security  Functionality 
Verification 

-  SI-7  Software  and  Information 
Integrity 

-  CA-5  Plan  of  Action  and 


Milestones 

-  RA-3  Risk  Assessment 

-  SA-2  Allocation  of  Resources 

-  SA-3  Life  Cycle  Support 

-  SA-4  Acquisitions 

-  SA-6  Software  Usage 
Restrictions 

-  SA-10  Developer  Configuration 
Management 

-  SA-1 1  Developer  Security 
Testing 

-  SA-1 2  Supply  Chain  Protection 

-  SA-1 3  Trustworthiness 

-  PM-1  Information  Security 
Program  Plan 

-  PM-3  Information  Security 
Resources 


-  PM-6  Information  Security 
Measures  of  Performance 

-  PM-7  Enterprise  Architecture 

-  PM-9  Risk  Management 
Strategy 

-  SA-8  Security  Engineering 
Principles 

-  CM-6  Configuration  Settings 

-  CM-  8  Information  System 
Component  Inventory 

-  SI-1 1  Error  Handling 

-  SI-12  Information  Output 
Handling  and  Retention 


Key  questions 


►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


Perspective  on  technology  today 


Technology  is  an  integral 
part  of  our  lives 
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Critical  Infrastructure  /  Key  Resources 


■  Ra^road  Tiaras 
Highway  5r dges 
■  Pipelines 

-  port* 
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Physical  Infrastructure 


Contra  -iy-E-twms 
■  SC  ADA 

-  PCS 

-  DCS 


Servces 

Managed  Security 
-  Information 
brvfcss 


Ha  id  ware 

-  Catahase  Servers 
NetworWng  Eq  u1  pment 


Software 

-  FfnancJai  Systems 
■  Human  R&m-uhc&e 


nternet 

Domain  Name  System 
-  Vi  Jet-  Hosting 


Cyher  Infrastructure 


IT  and  Communications 
products  are  assembled,  built, 
and  transported  by  multiple 
vendors  around  the  world 
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Malicious  actors  are  taking  advantage  of  abundant  opportunities  to 
tamper  with  and  sabotage  products  ... 


What 

commonalities 

exist? 

83%  of  victims  were  targets  of 
opportunity 

92%  of  attacks  were  not  highly 
difficult 

86%  were  discovered  by  a  third 
party 

96%  of  breaches  were  avoidable 
through  simple  or  intermediate 
controls 

How  do 

breaches 

occur? 

50%  utilized  some  form  of 
hacking 

49%  incorporated  malware 

(lower  percentages  included 
physical  attacks,  privilege  misuse, 
and  social  tactics) 

*  Source  -  2011  Verizon  Data  Breach  Investigations  Report 


SQL  Injection? 


Counterfeit? 

CRITICAL 


CRITICAL 


MM  mrer 


Counterfeit? 

q  m 


Logic  bomb? 
Tamper? 
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Counterfeit? 
Logic  bomb? 


...  ultimately  compromising  system  integrity  and  operations 
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SwA  requires  multi-disciplinary  collaboration 


Communication 

Challenges 

►  Vocabulary 

►  Reserved  Words 

►  Priorities 

►  Perspective 

►  Experience 

►  Objectives 

►  Drivers 

►  Risks 

Source:  https://buildsecuritvin.us-cert.gov/swa/procresrc.html 


Without  a  common  language  we  cannot  communicate  across  disciplines 


Acquirers  of  IT  products  and  services  trust  that  suppliers  are 
addressing  cyber  security  without  validating 


Acquisition  and  Outsourcing 


Initiate  an  Monitor  the  Acceptance  Operational 

agreement  agreement  System 


Acquisition  Practice 

47%  do  not  perform  acceptance  testing  of  third-party  code 

Product  Development  and  Maintenance 

Requirements  Management:  19%  do  not  carry  out  security 
requirement  definition 

Design/Develop:  27%  do  not  practice 
secure  design 

Test:  30%  do  not  use  static 
analysis/manual  code  review 

Key  questions 

►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


Communication  across  organizational  stakeholders  is  critical 
to  addressing  SwA  challenges 


Define  Business  Goal 


Development  Organization 

DO  1  Establish  the  assurance 
resources  to  achieve  key 
business  objectives 

DO  2  Establish  the  environment  to 
sustain  the  assurance 
program  within  the 
organization 


I* 


Development  Project 

DP  1  Identify  and  manage  risks 
due  to  vulnerabilities 
throughout  the  product  and 
system  lifecycle 

DP  2  Establish  and  maintain 

assurance  support  from  the 
project 

DP  3  Protect  project  and 

organizational  assets 


Prioritize 
funds  and 
manage  risks 


Enterprise  Assurance 
Support 

ES  1  Establish  and  maintain 
organizational  culture 
where  assurance  is  an 
integral  part  of  achieving 
the  mission 

ES  2  Establish  and  maintain  the 
ability  to  support 
continued  delivery  of 
assurance  capabilities 

ES  3  Monitor  and  improve 

enterprise  support  to  IT 
assets 


Acquisition  and  Supplier 
Management 

AM  1  Select;  manage,  and  use 
effective  suppliers  and 
third  party  applications 
based  upon  their 
assurance  capabilities. 


Development  Engineering 

DE  1  Establish  assurance 
requirements 

DE  2  Create  IT  solutions  with 
integrated  business 
objectives  and  assurance 

DE  3  Verify  and  Validate  an 
implementation  for 
assurance 


Enable 
Resilient 
Technology 


Sustained 
environment  to 
achieve 
business  goals 
through 
technology 


The  Assurance  PRM  Is  A  Holistic  Framework  that  connects  CMMI  and  RMM  to  facilitate  communication 

https://buildsecuritvin.us-cert.gov/swa/proself  assm.html 


A  majority  of  SwA  best  practices  focus  on  developer-centric 
audiences  from  a  security  point  of  view 


Development  Organization 

DO  1  Establish  the  assurance 

resources  to  achieve  key 
business  objectives 

DO  2  Establish  the  environment  to 
sustain  the  assurance 
program  within  the 
organization 


Development  Project 

DP  1  Identify  and  manage  risks  due  to 
vulnerabilities  throughout  the 
product  and  systemlifecycle 

DP  2  Establish  and  maintain 

assurance  support  from  the 
project 

DP  3  Protect  project  and 

organizational  assets 


Development  Engineering 

DE  1  Establish  assurance 
requirements 

DE  2  Create  IT  solutions  with 

integrated  business  objectives 
and  assurance 

DE  3  Verify  and  Validate  an 

implementation  for  assurance 


Enhancing 

the  Development  Life  Cycle 
to  Produce  Secure  Software 
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Executive  commitment  ->  SDL  a  mandatory  policy  at  Microsoft  since  2004 


Education 


Technology  and  Proca-ss 


Accountability 


Ongoing  Process  Improvements  ->  6  month  cycle 

h^UlmtYLmi*J932ft 


Assurance  for  CMMI  ® 
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Key  questions 

►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


100  apps  written  by  100  developers  at  100  companies 


►  83  apps  have  serious  vulnerabilities 

►  72  apps  have  cross  site  scripting 

►  40  apps  have  SQL  Injection 

►  100  apps  contain  code  of  unknown 
origin 

►  90  apps  use  un-patched  libraries  with 
known  flaws 

►  5  apps  have  had  a  scan  or  pentest 

►  1  app  has  had  a  manual  security  code 
review 

►  0  apps  provide  any  visibility  into  security 
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Why 

►  1  company  has  a  responsible  appsec 
program 

►  1  developer  has  any  security  training 


Adapted  from:  The  Open  Web  Application  Security  Project  Jeff  Williams,  Aspect  Security,  S1/14A  Forum  Sept  2010 


Implementation  lessons  learned  from  some  of  the  1/100 
companies  that  implement  SwA  successfully 


►  Who 

-  Secure  development  SMEs 

-  Developers 

►  What 

-  Measure  progress  (training,  secure  code  reviews, 
security  change  requests) 

-  Internal  policy 

-  During  product  development  process 

-  During  Leadership  discussions 

-  As  part  of  development  and  acquisition  reviews 

►  Where 

-  IT  Development  Organizations 

-  IT  Acquisition  Organizations 

-  IT  Integrator  Organizations 

Courtesy  of  September  2010  SwA  Panel  SwA  Practices 
-  Getting  to  Effectiveness  in  Implementation 


►  Why 

-  Customer  pressure 

-  Reaction  to  an  incident 

►  Why  Not 

-  Compliance  drivers  don’t  exist 

-  Focus  is  on  systems  and  networks 

-  Secure  software  training  is  not  given  to 
developers  and  architects 

►  How 

-  Executive  leadership  commitment 

-  Translate  ROI  to  project  manager  vocabulary 
(cost,  schedule,  quality) 

-  Start  small  and  build 

-  Use  coding  standards 

-  Empower  secure  development  to  prevent  a 
product  from  moving  to  the  next  milestone 

-  Avoid  creating  a  new  language 

-  Leverage  what  is  already  known 

-  Increase  automation  of  menial  tasks 
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Key  questions 

►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


Measure,  measure,  and  measure  again 


"  The  only  man  I  know  who 
behaves  sensibly  is  my  tailor; 
he  takes  my  measurements 
anew  each  time  he  sees  me. 
The  rest  go  on  with  their  old 
measurements  and  expect 
me  to  fit  them." 


-  George  Bernard  Shaw 


Source:  www.CartoonStock.com 


Efficiency/Effectiveness 


Robust  measurement  does  not  happen  overnight  and  requires 
foundational  capabilities  in  place  to  be  effective 


Goal 


Prepare  Develop  Capability  Use  Capability 

Foundation 


Metric  Type  Implementation 

Metric 


Effectiveness 

Metric 


Impact 

Metric 


LINCOLN  LABORATORY 

Massachusetts  Institute  of  Technology 


Courtesy  of  Matt  Coose,  DHS 


Critical  success  factor  -  long-term  management  commitment, 
focus,  and  appropriate  expectations 


Problem 


►  Senior  management  has  not  expressed  explicit 
support  for  the  program 

►  There  is  no  long  term  funding  and  resource 
commitment 

►  There  is  a  perception  in  the  organization  that 
measurement  and  monitoring  are  temporary  and 
“will  be  done”  at  some  point 

►  Security  improvements  are  expected  within  a  very 
short  time  period  and  are  expected  to  be  easily 
directly  related  to  measurement  and  monitoring 

►  Program  is  expected  to  be  perfectly  planned  and 
executed  as  if  the  organization  has  done  this  a 
million  times  and  has  the  process  down  perfectly 

►  Turnover  and  changes  in  roles  breaks  up 
continuity 

►  Accountability  for  metrics  is  difficult  to  assign 


Solution 


►  Obtain  senior  management  commitment  before 
you  start 

►  Work  across  the  organization  with  the 
stakeholders  to  make  them  a  part  of  the  solution 

►  Iterate  the  program  to  measure  critical  things 

►  Structure  the  program  to  begin  with  quick  wins  to 
continuously  demonstrate  increase  in  value 

►  Manage  expectations  continuously  -  explain  that 
the  long  term  focus  is  critical 

►  Assign  roles,  train  your  responsible  parties,  and 
communicate  that  continuity  is  key  for  success 

►  Emphasize  positive  reinforcement  -  if  everyone  is 
failing,  nobody  will  cooperate  and  the  program  will 
fail 
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Critical  success  factor  -  realistic  and  well  thought  out  data 
collection  strategy 


Problem 


►  Data  sources  not  well  known,  well  defined,  and 
therefore  not  considered 

►  Desired  data  is  not  obtainable 

►  Automated  data  sources  are  dispersed 
throughout  the  organization  and  data  is  difficult 
to  consolidate 

►  Authoritative  data  sources  do  not  exist 

►  Data  is  collected  inefficiently  or  incorrectly 

►  Collection  deemed  too  difficult  too  early 

►  Difficulty  or  inability  to  capture  historical  data 


Solution 


►  Identify  all  available  automated  and  manual 
data  sources 

►  Define  data  that  you  need  and  compare  with 
what  available 

►  Create  data  collection  strategy  that  would 
balance  the  need  for  data  with  the  current 
state  and  plan  for  deliberately  expanding  data 
sources  and  measures 

►  Define  future  changes  to  processes/tools  or 
requirement  for  new  tools  early  in  the  process 
and  refresh  as  you  learn  more 

►  Use  feasibility  of  data  collection  as  one  of  the 
criteria  for  metrics  selection 

►  Train  your  data  collectors  and  information 
owners  about  what  you  need  and  then  ask  for 
it  repeatedly 
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Critical  success  factor  -  effective  use  of  the  measures  to  improve 
security 


Problem 


►  Metrics  analysis  is  not  prioritized  according  to 
strategic  goals,  risks,  ROI,  or  other  explicit 
criteria  agreed  upon  by  the  organization 

►  Wrong  types  of  measures  are  distributed  to 
wrong  stakeholders 

►  Measures  are  collected  for  compliance  and  are 
not  used  to  improve  security 

►  Metrics  data  is  not  used  for  risk-based  decision 
making 


Solution 


►  Develop  criteria  for  prioritizing  and  scoring 
measures  early  in  the  program  and  reconsider 
every  time  you  expand  the  program 

►  Define  who  gets  what  data  and  reassess 
periodically  -  ask  your  customers  if  it  is  still 
useful 

►  If  metrics  are  not  used  for  decision  making  and 
improvement,  drop  them 

►  Communicate,  communicate,  communicate 


Security  control  measures 


►  Percent  of  new  systems  that  have  completed  certification  and  accreditation  (C&A)  prior  to  their 
implementation  (NIST  SP  800-53  Control:  CA-6:  Security  Accreditation) 

►  Percent  of  employees  who  are  authorized  access  to  information  systems  only  after  they  sign 
an  acknowledgement  that  they  have  read  and  understood  rules  of  behavior  (NIST  SP  800-53 
Controls  -  PL-4:  Rules  of  Behavior  and  AC-2:  Account  Management) 

►  Percent  of  the  agency’s  information  system  budget  devoted  to  information  security  (NIST  SP 
800-53  Controls  -  SA-2;  Allocation  of  Resources) 


Security  Control  Measures  address  compliance  with  the  end  state  of  the 
system,  but  not  the  underlying  processes,  structures,  and  code 


Measurement  for  secure  code  requires  understanding  code  level 
attributes  ... 


OCI 


Enumerations  and 
scoring  schemas 
provide  units  to  count 
and  therefore 
measure 


Courtesy  of  Bob  Martin 
MITRE  Carp 


nvd.nist.gov 


Measurement  for  secure  code  involved  understanding  the 
effectiveness  of  implemented  processes 


►  Acquisition 

-  Number  and  percent  of  acquisition  discussions  that  include  SwA  representative 

-  Number  and  percent  of  contracting  officers  who  received  training  in  the  security 
provisions  of  the  FAR 

-  Percent  of  documented  Supplier  claims  verified  through  testing,  inspection,  or  other 
methods 

-  Number  and  percent  of  relevant  high  impact  vulnerabilities  (CVEs)  present  in  the  system 

►  Testing 

-  Number  and  percent  of  tests  that  evaluate  application  response  to  misuse,  abuse,  or 
threats 

-  Number  and  percent  of  tests  that  attempt  to  subvert  execution  or  work  around  security 
controls 

-  Percent  of  untested  source  code  related  to  security  controls  and  SwA  requirements 


How  to  apply....  success  is  simple 


Get  Management  Support 
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Start  Small 


Measure  Behavior 


•  Scope  out  measurement 

•  Obtain  tangible  support  for 
security  measures 
development  and  use  at 
every  management  level 

•  Maintain  support  through 
regular  reporting  to 
stakeholders,  tailored  to  their 
levels 

-Address  their  goals 

Refine  detail  further  up  the 
management  chain 

-Use  Dashboards  & 
Reports  to  endure 


•  Expand  your  project/program  cost,  schedule, 
quality,  and  growth  measures  to  cover 
security 

•  Start  with  a  manageable,  small  set  of 
security  measures  adding  more  measures  as 
the  project  matures 

•  Leverage  existing  industry  lists  and  select 
applicable  measures 

•  Use  a  framework  to  translate  measures  from 
industry  lists  into  your  organization’s 
approach 

•  Train  data  collectors  to  think  in  terms  of 
metrics 


•  Identify  and  measure  best 
and  worst  process  and 
practice  behaviors  as  well 
as  results 

•  Create,  display,  and  report 
measures  to  influence 
appropriate  behavior 

•  Take  advantage  of 
unintended  consequences 
produced  by  measurement 

•  Reuse  measures  where 
possible 


Incorporate  security  measures  into  your  existing  measurement  activities 

Source:  http://www.psmsc.com/Downloads/TechnoloqyPapers/SwA%20Measurement%2010-08-08.pdf.  accessed  4/10/09 
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Key  questions 

►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 


►  Next  Steps? 


Service 


Business  functions  rely  on  accurate  and  reliable  information  from 
technology  that  functions  as  intended  (and  only  as  intended) 


'CEOT 


Software  Engineering  Institute 


Carnegie  Mellon 


Adapted  from:  November  2009  SwA  Forum-Evolution  in  SwA  Processes  Panel  -  David  White,  SEI 
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Potential  impacts  from  threats  to  business  functions  can  be 
understood  by  communicating  software  level  vulnerabilities 


How  do  we  prevent 
this  next  time? 


Are  we  being 

_ attacked? 


Controls 


Applied  to 


Asset 

Management 


Are  we  at 
risk? 


Design  V 

( ) 

P\ar\y 

Develop  ^ 

Acquire  ^ 

Deploy  y  j 

Decommission 


Adapted  from  September  2010  SwA  Forum, 
CERT  RMM  for  Assurance  ,  Lisa  Young,  SEI 


Who  is  attacking  and 
what  do  they  want? 


Enterprise  Ass  urance 
upport 

ESI  Establish  and  maintain 
organizational  culture 
where  assurance  is  an 
inteqral  part  of  achieving 
the  mission 

ES  2  Establish  and  maintain  the 
ability  to  support 
continued  delivery  of 
assurance  capabilities 

ES  3  Monitor  and  improve 

enterprise  support  to  IT 
assets 


http://www.ruggedsoftware.org/ 


Tti«  Ruga61*  Software 


Manifest0 
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Key  questions 

►  Who  asks  developers  to  develop  secure  code? 

►  Do  developers  know  why  they  need  to  develop  secure  code? 

►  What  should  developers  do/not  do  to  develop  secure  code? 

►  Can  developers  develop  secure  code  by  themselves? 

►  How  do  developers  know  they  have  developed  secure  code? 

►  Why  should  developers  care? 

►  Next  steps? 


Cyber  security  and  software  assurance  standard  development 
organization  landscape 


Other  Organizations 


Technical  Committees^1 
Other  Standards  Bodies 


Systems  Audit*' 
ComrolAuoa«Mn 


The  Open 
Group 


Common  Osena 
Dmetopment  BoenJ 


QuEST  Fonri 


s 

t.  .  s 

oott 

«cum 

n  A 

hr  ( 
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SC22  -  Programming  Languages,  ISO/IEC  TR  24772,  Programming 
Language  Vulnerabilities 


►  Targets  building  software  that  is  inherently  less  vulnerable  through  improving  the  programming 
languages,  or,  at  least,  improve  the  usage  of  them  in  coding 

►  A  catalog  of  60+  issues  that  arise  in  coding  when  using  any  language  and  how  those  issues 
may  lead  to  security  and  safety  vulnerabilities 

►  Cross-referenced  to  CWE 

►  Each  discussion  includes 

-  Description  of  the  mechanism  of  failure 

-  Recommendations  for  programmers:  How  to  avoid  or  mitigate  the  problem. 

-  Recommendations  for  standardizers:  How  to  improve  programming  language  specifications. 


Courtesy  of  Jim  Moore,  MITRE 


ISO/IEC  27036:  Information  technology  -  Security  techniques  - 
Information  Security  for  Supplier  Relationships 


►  Scope:  This  international  standard  covers  information  security  in  relationships  between 
acquirers  and  suppliers  to  provide  appropriate  information  security  management  for  all  parties. 
In  particular,  it  also  includes  management  of  information  security  risks  related  to  these 
relationships. 

►  The  standard  will  be  subdivided  into  the  following  parts: 

-  Part  1  -  Overview  and  Concepts 

-  Part  2  -  Common  Requirements 

-  Part  3  -  Guidelines  for  ICT  Supply  Chain 

-  Part  4  -  Guidelines  for  Outsourcing 


NIST  IR  7622,  Piloting  Supply  Chain  Risk  Management  for 
Federal  Information  Systems 


►  Initially  based  on  DoD  ICT  SCRM  Key  Practices  document  and  developed  in  close 

collaboration  with  the  industry 

►  Introduces  the  notion  of  supply  chain  players 

-  Acquirer  -  For  this  document,  the  acquirer  is  always  a  government  agency  (including  those 
agencies  taking  on  the  role  of  integrator). 

-  Integrator  -  A  third-party  organization  that  specializes  in  combining  products/elements  of 
several  suppliers  to  produce  elements  (information  systems). 

-  Supplier- Third-party  organization  providing  individual  elements.  Synonymous  with  vendor 
and  manufacturer;  also  applies  to  maintenance/disposal  service  providers 

►  Lays  out  pre-requisites  of  being  able  to  address  ICT  SCRM  challenge 

►  States  specific  practices  that  are  consistent  with  DoD  guidance  and  ISO  frameworks 


SAFECode  (www.safecode.org) 


►  SAFECode  is  a  global,  industry-led  effort  to  identify  and 
promote  best  practices  for  developing  and  delivering 
more  secure  and  reliable  software,  hardware  and 
services 

►  White  papers 

-  Software  Assurance:  An  Overview  of  Current 
Industry  Best  Practices 

-  Fundamental  Practices  for  Secure  Software 
Development 

-  Security  Engineering  Training:  A  Framework  for 
Corporate  Training  Programs  on  the  Principles  of 
Secure  Software  Development 

-  Framework  for  Software  Supply  Chain  Integrity 

-  Software  Integrity  Controls:  An  Assurance-Based 
Approach  to  Minimizing  Risks  in  the  Software  Supply 
Chain 


The  Open  Group 

Trusted  Technology  Provider  Framework  (TTPF) 

Purpose 

Identify  and  gain  consensus  on  common  processes,  techniques,  methods,  product  and  system 
testing  procedures,  and  language  to  describe  and  guide  product  development  and  supply  chain 
management  practices  that  can  mitigate  vulnerabilities  which  could  lead  to  exploitation  and 
malicious  threats  to  product  integrity. 

Objectives 

•  Identify  product  assurance  practices  that  should  be  expected  from  all  commercial 
technology  vendors  based  on  the  baseline  best  practices  of  leading  trusted  commercial 
technology  suppliers 

•  Help  establish  expectations  for  global  government  and  commercial  customers  when  seeking 
to  identify  a  trusted  technology  supplier 

•  Leverage  existing  globally  recognized  information  assurance  practices  and  standards 

•  Share  with  commercial  technology  consumers  secure  manufacturing  and  trustworthy 
technology  supplier  best  practices 

•  Harmonize  language  used  to  describe  best  practices 

Source:  Source:  September  28,2010  SwA  Forum,  DoD  Trusted  Defense  Systems,  Ms.  Kristen  Baldwin,  DDR&E/Systems  Engineering 
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What’s  next? 


►  Continued  collaboration  to: 

-  Reach  and  enable  developers 

-  Reach  and  enable  executives 

-  Develop  and  promote  resources  for  us  by  developers  and  executives 

►  Participation  in  international  standardization  efforts 

-  SC7  TAG  intersections  through  your  SC7  TAG 

-  CS1/SC27 

-  IEEE  representative  to  the  SC7  TAG 

-  SC22 

►  Participation  through  the  SwA  Working  Groups  and  Forum 


►  Stay  Tuned  ... 


Contact  information 


Nadya  Bartol 

Senior  Associate 

Booz  |  Alien  j  Hamilton 

Booz  Allen  Hamilton  Inc. 
One  Preserve  Parkway 
Rockville,  MD  20852 
Tel  (301)444-4114 
bartol_nadya@bah.com 

Michele  Moss 

Lead  Associate 

Booz  |  Alien  j  Hamilton 


Booz  Allen  Hamilton  Inc. 
8283  Greensboro  Drive 
Mclean,  VA  22102 
Tel  (703)377-1254 
moss_michele@bah.com 
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Presentation  goal 

Background  about  SPA  WAR  PACIFIC 
Current  situation 
Improvement  focus 
Challenges 
Approach 
Questions 


SHMR 

Systems  Center 
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SPAWAR  is  the  Navy  expert  in  delivery  and  sustainment  of  C4ISR  systems 

SPAWAR  =  Space  and  Naval  Warfare  Systems  Center  C4ISR  =  command,  control, 
communications,  computers,  intelligence,  surveillance,  and  reconnaissance 
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♦•*«■  if 


•  How  the  organization  utilized  Lean  and  Six  Sigma®  process  improvement 
techniques  to  affect  implementation  of  CMMI®  practices 

-  Occurs  during  simultaneous  organizational  realignment 

-  Involves  consolidation  of  five  major  service-oriented  projects 

Focused  on  early  improvements  for  the  individuals  working  on  the  projects  (In  the 
trenches) 

Assisted  with  institutionalizing  selected  targeted  practices  utilizing  some  Level  III 
practices 


SPAWAR 

_ ® 

Y 

Systems  Center 
PACIFIC 


Six  Sigma  is  a  registered  trademark  of  Motorola  Trademark  Holdings,  Inc.  in  the  U.S.  and/or  other 
countries.  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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•  DOD  U.S.  NAVY  Organization 

More  than  4,000  scientists  and 
engineers 

Located  in  San  Diego  and 
throughout  the  globe 

•  INDUSTRY  PATTERNS  DOD 
U.S.  NAVY  Organization 

More  than  4,000  scientists  and 
engineers 

Located  in  San  Diego  and 
throughout  the  globe 


SPAWAR 
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Systems  Center 
PACIFIC 


SPAWAR  is  the  Navy  expert  in  delivery  and  sustainment  of  C4ISR  systems 

SAIL 
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•  Design,  build,  and  sustain  C4ISR 
information  dominance  systems 


(Radar,  networks,  command  and 
control,  crypto  devices,  satellite 
communications,  submarines 
electronic  systems,  etc.) 


SHMR 
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SPAWAR  is  the  Navy  expert  in  delivery  and  sustainment  of  C4ISR  systems 

SAIL 
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Reliability 


Availability 


Maintainability 
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SPAWAR  is  the  Navy  expert  in  delivery  and  sustainment  of  C4ISR  systems 

SAIL 


6 


©  SAIC.  All  rights  reserved. 


Implemented  Software  (SW)  Capability  Maturity  Model^ 
CMM  ®  predecessor  of  CMMI®  model. 

Systems  Engineering  Process  Office  (SEPO) 


Attained  SW-CMM  Level  3  in  October  2000. 

SSC  PAC  transited  from  SW-CMM  to  CMMI-  DEV  model 
and  continues  with  its  process  improvement  road. 


Implementing  CMMI-SVC  ML2  model  1.2  for  Services 
projects 

Achieve  CMMI-DEV  ML3  on  2012 


SrWAR 

Systems  Center 
PACIFIC 


SPAWAR  is  the  Navy  expert  in  delivery  and  sustainment  of  C4ISR  systems 

Capability  Maturity  Model,  CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University. 
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Different  Levels  of  Existing  Consolidation 


srmm 


Systems  Center 
PACIFIC 


ISNS  =  Integrated  Shipboard  Network  Systems  CENTRIXS  =  Combined 
Enterprise  Regional  Exchange  System  SCI-N  =  Sensitive  Compartmented 
Information  Networks  ADNS  =  Automated  Digital  Network  System  SubLAN  = 
Submarine  Local  Area  Network 
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•  Standard  organizational  change  resistance 

•  Limited  budget  to  accomplish  improvements  and  certification 

•  Individual  participant  perception  that  they  were  being  asked  to  do  more  with 
limited  or  no  additional  funding 

-  Consolidating  existing  processes 

Addressing  gaps  in  CMMI®  practices  that  weren’t  currently  being  accomplished 
Perception  that  improvement  benefits  weren’t  going  to  be  easily  or  quickly  realized 

•  Revolving  individual  roles  and  responsibilities 


SPAWAR 

Systems  Center 
PACIFIC 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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•  Defined  the  “organization”  as  part  of  this  initial  effort  as  the  five  projects  while 
monitoring/participating  in  the  wider  organization  (SSC  PAC)  CMMI®  efforts 

•  Develop  and  utilize  draft  Organizational  Process  Focus  (OPF),  Organizational 
Process  Definition  (OPD)  and  Integrated  Project  Management  (IPM)  practices 
from  the  very  beginning  to  support  aligning  five  projects.  Pursued  a 
continuous  versus  a  staged  approach  for  the  five  projects. 

•  Simultaneous  improvement  emphasis  on  the  project  planning  and  project 
monitoring  and  control  processes  so  that  all  practitioners  (project  managers 
and  functional  leads)  are  performing  in  a  similar,  integrated  fashion 


srmm 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Improvement  Focus 


▼ 

Systems  Center 
PACIFIC 


ISNS  =  Integrated  Shipboard  Network  Systems  CENTRIXS  =  Combined  Enterprise  Regional 
Exchange  System  SCI-N  =  Sensitive  Compartmented  Information  Networks  ADNS  = 
Automated  Digital  Network  System  SubLAN  =  Submarine  Local  Area  Network 
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After  Before 


Improvement  Focus  on  Effort  for  a  Particular 
Process  Area 
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CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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•  Apply  Lean  Six  Sigma®  (LSS)  improvement  efforts  to  establish  and/or  improve 
the  “How”  of  the  focused  project  processes,  making  sure  that  you  incorporate 
the  gaps  between  the  existing  processes  and  the  CMMI®  best  practices 

•  Simultaneously 

-  Apply  Lean  efforts  to  remove  waste  in  current,  focused  project  processes 

Apply  Six  Sigma  efforts  to  improve  the  quality  and  reduce  the  rework  of  current 
focused  processes 

Ensure  that  the  improvement  phase  incorporates  the  gaps  and  adapts  CMMI 
practices  to  the  business  environment,  thus  improving  the  focused  processes 


SPAWAR 

V 

Systems  Center 
PACIFIC 


Six  Sigma  is  a  registered  trademark  of  Motorola  Trademark  Holdings,  Inc.  in  the  U.S.  and/or  other  countries. 
CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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•  Get  really  close  to  the  way  the  various  personnel  (practitioners)  operate  to 
apply  the  improvements  and  improve 

To  adopt  practices  practitioners  need  to  be  able  to  accomplish  their  work  faster 
and  with  better  quality  and  less  rework 

•  Rather  than  adopt  new  technology  (tools),  we  focused  on  improvements  in 
the  use  of  existing  technology 

This  puts  more  focus  on  training  and  education  as  opposed  to  increased  cost  of 
acquiring  new  tools,  adopting  to  existing  processes,  and  conducting  training 

Practitioners  are  more  comfortable  with  using  existing  tool  sets  but  in  improved 
ways  as  opposed  to  adopting  newer,  unfamiliar  tool  sets 

•  Design  the  new  way  (improvement)  to  doing  business,  ensuring  that  it 
incorporates  the  “What”  of  CMMI®  practices  for  the  particular  process  area 


srmm 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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•  Accomplished  Level  II  certification  on  time  for  all  five  projects 

•  Continued  to  apply  the  improvement  technique  to  mature  the  projects  toward 
Level  III  selected  process  areas  in  a  continuous  approach 
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Diagnosis  PIF: 
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Improvement  Fatigue 
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“Ugh,  not  another  process  change! 
Didn’t  we  just  have  one  last  month?” 
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Diagnosis:  PIF 


Treatment 
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Why  Are  We  Doing  This? 
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The  People  Factor 
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Prescription  for  PIF 
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If  All  Else  Fails 
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Contact  Info: 

Craig  Hale 

Process  Improvement  Manager 
Esterline  Control  Systems  -  AVISTA 
Phone  (608)348-8815 
Fax  (608)348-8819 
Email  Craig.Hale@esterline.com 
www.  este  r  I  i  n  e .  co  m/co  n  tro  I  sy  ste  m  s/a  v  i  sta 


to  Embrace  People 
in  an  Agile  Approach 

and 

Support  High  Maturity 

(OPM) 


Definitions 
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Agile  -  Agile  methods  emphasize  face-to-face 
communication  over  written  documents  when  the  team  is 
all  in  the  same  location 

OPM  -  The  purpose  of  Organizational  Performance 
Management  (OPM)  is  to  proactively  manage  the 
organization’s  performance  to  meet  its  business  objectives 


I  Specific  Goal  and  Practice  Summary 

SG  1  Manage  Business  Performance 

•  SP  1.1  Maintain  Business  Objectives 

•  SP  1 .2  Analyze  Process  Performance  Data 

•  SP  1.3  Identify  Potential  Areas  for  Improvement 

SG  2  Select  Improvements 

•  SP  2.1  Elicit  Suggested  Improvements 

•  SP  2.2  Analyze  Suggested  Improvements 

•  SP  2.3  Validate  Improvements 

•  SP  2.4  Select  and  Implement  Improvements  for  Deployment 

SG  3  Deploy  Improvements 

•  SP3.1  Plan  the  Deployment 

•  SP3.2  Manage  the  Deployment 

•  SP  3.3  Evaluate  Improvement  Effects 


High  Maturity  -  focuses  on  continually  improving  process 
performance  through  incremental  and  innovative  process 
and  technological  improvements.  The  organization’s 
quality  and  process  performance  objectives  are  established, 
continually  revised  to  reflect  changing  business  objectives 
and  organizational  performance,  and  used  as  criteria  in 
managing  process  improvement.  The  effects  of  deployed 
process  improvements  are  measured  using  statistical  and 
other  quantitative  techniques  and  compared  to  quality  and 
process  performance  objectives 
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The  Solution  We  Chose 


Strategic  Policy 
Deployment 
(SPD) 


Strategic  Policy  Deployment 


Combination  of: 


Clear  &  Aligned  Priorities 
Behavior  Changes 
Change  in  Thinking  (PDCA) 
Elimination  of  Waste 


. .  .to  achieve  Business  Results 


Strategic  Policy  Deployment 


A  process  to  focus  upon  Goals,  that  cut  across  the 
corporation 

Aligns  &  links  resources  &  action  in  pursuit  of  those 
Goals. 

Enables  progress  towards  the  Goals  to  be  measured 

Enables  rapid  root  cause  corrective  action  if  results 
vary  from  goals 

Drives  process  improvement 

Individuals  &  teams  get  clarity  on  their  impact  upon 
the  Goals 

It  becomes  the  yearly  implementation  of  our  long  term 
strategic  planning  process. 


Policy  Deployment  as  a  Tool 


Deployment  is  an  effective  tool  to  use  for  answering  the  following 
questions: 

•  How  do  we  identify  our  critical  goals? 

•  How  do  we  develop  plans  and  align  our  activities? 

•  How  do  we  communicate  our  goals  and  activities  level  by  level? 

•  How  do  we  align  the  abundant  talent  of  our  team  members  on  the 
critical  few? 

•  How  do  we  sustain  our  activities? 

•  How  do  we  quickly  change  course  when  required? 

•  How  do  we  learn  from  our  experience? 


Magnitude  of  Change 

Behavior  Change 

•  Discipline 

•  Emphasis  on  how  the  organization  will  deliver  the 
priorities 

•  Catchball  to  understand  the  priorities  and  the  means  to 
deliver  them 

•  Gemba  -  look  for  evidence  the  plan  is  proceeding  and 
in  control 

Clear  and  Aligned  Priorities 

•  Start  with  top  management  priorities  and  link/translate 
at  every  level 

•  Critical  few  metrics  match  Excel  commitments 

•  Must  deselect 
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Step  1 

Choose  the  Focus 


Check 


Step  2 

Align  the  Organization 


Step  3 

Implement  the  Plan 


Operational 

Focus 


Stefjp^cfJiently 

Review  and  Improve 


MBF, 

Newspaper, 
or  problem 
strip  for  red 
items 


Program  Plan  (multiple) 


jr  Process  to  build  consensus  through  dialog  about  the 
|  f  goals  and  how  to  achieve  them. 

| !  Two  way  communication  that  arrives  at  a  collective 
wisdom  on  the  priorities  and  the  plans  to  deliver  the 
Bi  results. 

1  Leader  needs  to  have  a  vision  of  what  is  needed  and 
how  it  may  be  achieved. 

•  Team  will  provide  input  on  the  specific  how. 

A  •  The  leader  will  confirm  the  plan: 

Push  the  team  to  stretch  further  if  the  plan  comes  short  of 
what  he  had  in  mind. 

/]  Question  and  develop  understanding  of  the  plan  if  the  plan 

jCK  exceeds  what  he  had  in  mind. 


T  argets 


II  priorities  require  a  target  so  they  can 
be  measured. 


Targets  have  to  be  achievable,  challenging, 
based  on  reliable  data,  and  SMAR" . 


S  -  specific 


R  -  realistic 


-  timed 


M  -  measurable 
A  -  agreed 


The  DO  Phase 


Check 


Review  the  plan  on  a  regular  basis. 

•  Look  at  metrics  daily/weekly 

•  Formal  reviews  monthly 

results  vs.  expected,  as  well  as  the  countermeasure  to 
fill  the  gap 

Ask  why  if  the  team  is  doing  things  that  are  not  in  the 
plan. 

Question  frequently  by  going  to  see. 

•  Schedule  the  time  to  look  for  evidence  that  the  plan 
proceeding  and  in  control. 


w*m 


,  The  CHECK  Phase 

?■ 

[g  reviews  maintains  the  discipline  of  the 


Chec 


A  mini  PDCA  cycle  takes  place  everyday  as  activities 
are  checked  constantly 


Counter  measures  are  data  driven  looking  at 
root  cause. 

Check  if  previously  identified  counter 
measures  are  working  and  on  track. 

Don’t  react  to  noise. 

Escalate  issues  that  can  not  be  resolved  to 
the  next  level. 
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The  CHECK  Phase 


The  CHECK  Meeting 


manager  runs  the  review  meeting. 


The  manager  should  focus  on  standard  work: 
•Metrics  Chart 
•Action  Plan 

•Corrective  Action  for  RED  Items 
•Key  Items  at  Risk 


•  Asking  clarifying  questions  during  the  review 
process. 

•  Making  sure  that  each  person  knows  what  is 
expected  of  them  to  move  forward. 

•  Conf  irmation  check  -  will  this  plan  get  us 
there? 

•  Follow  up  at  Gemba  before  the  next  review  for 
key  issues. 


Company 


SPG/Unit 


Department 


Work  Team 


wer  levels  may  check  more  frequently. 


Check  Questions 


Policy  Deployment  -  Questions  to  Ask 
Do  you  have  a  plan? 

Does  the  plan  close  the  gaps  to  the  goal? 
Is  the  plan  being  executed  on  time? 

£>  Is  tthenpkm  rgen@mtitt^eth®ee^5ieet@d  measure 

business  results? 


Monitor  effectiveness  of  countermeasure 


Look  for  trends  -  not  just 

red/green 


•  Metrics  chart  gives  an  overall  view 
if  on  track 

•  Use  graphs  and  charts  to  see 
what  is  really  happening. 


The  ACT  Phase 


ft  off  track: 

•  Review  the  plan  and  countermeasures  to  confirm  that  gap 
from  the  plan  can  be  closed. 

Is  the  plan  itself  valid  or  does  it  need  to  be  modifid? 

?  •  Follow  up  to  determine  if  actions  on  the  countermeasures 
have  been  done. 

•  Go  to  GEMBA  to  check  to  see  if  progress  is  being  made 
on  critical  issues 

If  on  track 

•  Lock  in  the  condition  with  standardized  work 
Confirmation  Check 

•  Does  the  plan  and  countermeasures  link  to  the 
goal/vision? 

•  Does  it  seem  reasonable? 


Check 


On  Time  R&D  Schedule 


On  Quality  R&D 


On-Spec  to  book:  R&D  rework  cost 


On-budget  R&D 


On-time  R&D  milestones 


Operating  Profit 


Customer  Satisfaction  scores 
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Measures  &  GAPS 
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Policy  Deployment  Improvement  T arget  Countermeasure  Analysis 


What  is  the  problem  that 
is  preventing  us  from 
being  on  target? 


Data  Graph  1 


Data  Graph  2 


What  analysis  has  been 
done  to  determine  the  root 
cause  of  the  problem? 


TARGET 


'rioritization  &  Root  Cause 


Counter  Measures  &  Activities 


When 


Status/Results 


Predictive  Impact 


Kobi .  Vider(a)hotmail.  com 


Phone:  +972522946676 


Are  Y our  Customers  Happy? 

Using  customer  satisfaction  to  drive 

improvement  efforts 


Sue  Lafortune,  NS  A  Way  Program  Manager 
Kathy  Demery,  NS  A  Way  Infrastructure  Lead 
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Who  is  NSA? 

•  The  National  Security  Agency  was 
established  by  President  Harry  S. 
Truman  on  November  4, 1952. 

•  NSA  is  a  Defense  Agency  within  the 
US  Department  of  Defense  and  an 
Intelligence  Agency  within  the  US 
Intelligence  Community. 


-  By  original  charter,  the  Director  of  NSA  (DIRNSA)  is 
a  general  officer  grade  0-9  or  higher  selected  on  a 
rotating  basis  from  the  US  Army,  US  Air  Force,  US 
Navy. 

-  By  original  charter,  the  Deputy  Director  of  NSA 
(D/DIR)  is  always  a  DoD  civilian  employee. 
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What  does  NSA  do? 


•  NSA's  core  missions  are  to  protect  U.S. 
national  security  systems  and  to  produce 
foreign  signals  intelligence  information: 

-  Information  Assurance 

-  Signals  Intelligence 

-  Network  Warfare 


Information  Assurance  Mission 


The  information  assurance  mission 
confronts  the  formidable  challenge  of 
preventing  foreign  adversaries  from 
gaining  access  to  sensitive  or  classified 
national  security  information. 


Signals  Intelligence 


The  signals  intelligence  mission  collects, 
processes,  and  disseminates  intelligence 
information  from  foreign  signals  for 
intelligence  and  counterintelligence 
purposes  and  to  support  military 
operations. 
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Network  Warfare 


NS  A  enables  Network  Warfare  operations  to 
defeat  terrorists  and  their  organizations  at 
home  and  abroad. 


President  Barack  Obama,  Vice  President  Joe  Biden  and  members  of  the  National  Security  Team 
receive  an  update  on  the  mission  against  Osama  bin  Laden  in  the  Situation  Room  of  the  White 
House  on  May  1, 2011.,  Pete  Souza  /  Courtesy  of  the  White  House 
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NSA's  Mission  and  Systems 

Development 

•  The  Information  Assurance,  Signals  Intelligence, 
and  Network  Warfare  missions  are  highly 
technical. 

-  Systems  development  and  integration  —  particularly 
software-intensive  systems  —  are  a  key  enabler  in 
fulfilling  these  missions. 

•  NSA  has  a  diverse  set  of  software  development 
projects. 

•  Facilitating  timely  software  development  that 
delivers  quality  sufficient  to  fulfill  the  mission 
are  of  great  interest  to  NSA. 
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NS  A  Way  Overview-1 

•  "The  NS  A  Way  is  a  unified  framework  for  building 
large,  complex,  primarily  software  systems  that 
meet  the  diverse  needs  of  NSA  missions.  It  is 
lightweight,  intuitive,  and  independent  of  project 
size  and  development  methodology." 

•  It  is: 

•  Based  on  a  Customer/ Supplier  theme 

•  Focused  on  outputs  over  processes 

•  About  continuous  improvement 

•  Applicable  in  Agile,  Iterative,  and  Waterfall  LCMs 

•  Independent  of  team  size 
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NS  A  Way  Overview-2 

•  NSA  Way  defines  core  expectations  and  best  practices  for 
software  development. 

•  NSA  Way  deploys  'coaches'  into  NSA  systems  and  software 
development  organizations  to  provide  implementation 
guidance  and  to  assess  progress. 

•  NSA  Way  is  implemented  through: 

—  Gates  (Life  cycle  control-milestones) 

—  Metrics  (Quality  related) 

—  Processes  (currently  there  are  8) 

•  Driving  cultural  and  behavioral  change  first,  process 
maturity  second. 

•  NSA  Way  is  results  oriented. 
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Why  Customer  Satisfaction  at  NSA? 

Customer  satisfaction  surveys  measure  and  evaluate 
the  attitudes,  opinions  and  satisfaction  levels  of  your 
customers  and  clients. 

For  NSA  it  is  to.... 

-  understand  what  is  important  to  our  customers 

-  focus  on  what  we  can  do  to  meet  their  needs 

-  build  true  customer  understanding  and  knowledge 

-  drive  our  improvement  efforts 
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Why  Customer  Satisfaction  at  NSA? 


For  NSA  it  is  to... 


NOT  to... 


-  Increase  mission 
effectiveness,  customer 
efficiency,  realization  of 
National  Security 

-  Have  repeat  methods 
to  increase  mission 
agility 

-  Increase  National 
intelligence  value  and 
National  well  being 


-  increase  profits, 
customer 
loyalty,  brand 
recognition 

-  have  repeat 
business  to 
increase  market 
share 

-  increase 
earnings  and 
stock  profits 
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NSA's  Customer  Focused  Software 
Engineering  Approach  to  Integrated  PI 

•  Flexibility  of  NSA  Way 

•  Just  enough  process 

•  Focus  on  results  not  on  process 
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Customer  Driven  Priorities 


•  Need  to  know  what  is  most  important  to  our 
customer  (e.g.,  capabilities,  timeliness,  quality) 

•  Customer  priorities  are  different  for  every 
project 

•  Vary  the  rigor  of  best  practices  to  affect  the  end 
result 

•  Not  process  for  process  sake 
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This  is  What  We  Did 

•  Developed  a  customer  satisfaction  survey  service 

-  Developed  customer  satisfaction  survey 
questions 

—  Developed  two-dimensional  survey  method 
—  Researched  and  obtained  a  survey  tool 

•  Developed  and  conducted  training 

•  Piloted  our  Customer  Satisfaction  Survey  Service 

^  Customer  Satisfaction  Survey  is  just  the  beginning.  Next, 
need  to  analyze  results  and  develop  action  plans. 
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Customer  Satisfaction  Survey  Service 


Execution 

j  Analysis  & 

Communicate 

Continuous 

1  Conclusion 

Results 

Improvement 

Legend 


Project 
NSA  Way 
Project/NSA  Way 


Optimal  survey  window  is  within  30  to  90  days 
following  release/  deployment. 
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Customer  Satisfaction 
Survey  Questions  (Statements) 


Satisfaction  Area 

Statement 

Mission/ 
Business  Value 

The  capabilities  delivered  provided  the 
mission/business  value  expected. 

Product 

The  capabilities  delivered  were  as 
expected. 

Timeliness 

The  capabilities  were  delivered  when 
needed. 

Quality 

The  capabilities  delivered  met  my 
quality  expectations. 

Communication 

The  level  of  communication  throughout 
the  development  and  delivery  of  the 
capabilities  was  as  expected.  w 

Two-dimensional  Survey  Method 


Degree  of  Satisfaction 

Degree  of  Importance 

Very  Satisfied 

Very  Important 

Satisfied 

Important 

Neutral 

Neutral 

Dissatisfied 

Unimportant 

Very  Dissatisfied 

Very  Unimportant 

Unknown 

Unknown 

X  Knowing  the  customer's  degree  of  importance  helps 
prioritize  PI  efforts. 


Case  Study /  Lessons  Learned-1 

•  Response 

-  Low  response  rate  (~25%) 

-  Sample  size 

-  Not  statistically  significant 

•  Gathering  data 

-  Other  options  for  gathering  Customer  Satisfaction  data 

-  Customer  Satisfaction  Survey  not  be  confused  with 
other  "feedback"  (requirements  gathering) 

•  Customers 

-  Cherry  picking  the  customers 

—  Sampling  your  customer  set  (to  prevent  burnout) 

—  Anonymity 

—  Considerations  for  different  types  of  customers 
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Case  Study /  Lessons  Learned-2 


•  Survey  tool 

—  Steep  learning  curve 
—  Difficult  to  use 

•  Customer  Satisfaction  Survey  Service 
—  Life  cycle  too  long  (~  7  weeks) 
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Example:  Results 


Mission  Value: 


Timeliness: 


The  capabilities  delivered  provided  The  capabilities  were  delivered 
the  mission  value  expected.  when  needed. 
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Value  of  the  Comments. 


Numbers  are  one  aspect,  comments  are  another 

IT'S  LIKE  GOLD! 


Example  Comments: 

>  "Works  about  as  well  as  can  be  expected  for  a  web  app." 

>  "The  team  is  OUTSTANDING  when  it  comes  to 
responding  to  questions  or  when  I  am  in  need  of 
assistance.  Honestly,  they  respond  quicker  than  any 
other  database  support  team  that  I've  dealt  with  in  the 
past  seven  years.... 

>  "I  feel  that  this  organization  provides  quality  products 
and  results.  I  am  a  former  Z  product  user  and  since 
discovering  this  product  shortly  over  a  year  ago,  I  have 
no  further  use  for  the  Z  product,  as  the  quality  of  this 
product  is  much  higher  than  that  of  the  Z  product." 
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We  Survey  Our  Customers,  Too 


As  an  improvement  group,  we  need  to  know  our 
customers'  satisfaction  with  our  products  and 
services. 

•  Products 

-  Website 

-  Process  Assets 

•  Services 

-  Training 

-  Coaching 

-  Customer  Satisfaction  Survey  Service 

•  Modeling  the  behavior 
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Tips  for  Successful 
Customer  Satisfaction  Surveys-1 

•  Increase  sample  so  that  the  information  received 
is  statistically  significant. 

•  If  the  sample  size  is  small,  look  at  other  survey 
methods  (e.g.,  face  to  face). 

•  Survey  a  certain  percentage  over  a  period  of 
time  or  number  of  releases. 

•  Adjust  guidance  on  question  considerations 
based  on  relevance  to  type  of  customer. 

•  Provide  feedback  to  the  customer  early. 
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Tips  for  Successful 
Customer  Satisfaction  Surveys-2 

•  Increase  your  response  rate  -  Before  the  Survey 

-  Announce  the  upcoming  survey 

—  Have  upper-management  promote  &  encourage 
participation 

-  Survey  timing  (mid-week,  avoid  holidays) 

-  Clear,  easy,  and  quick  to  answer  questions 

-  Length  of  survey  is  short  (<10-15  questions) 

-  If  sent  survey  previously,  mention  your 
improvements  from  the  previous  survey 
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Tips  for  Successful 
Customer  Satisfaction  Surveys-3 

•  Increase  your  response  rate  -  During  the  Survey 

-  Attempt  contact  with  ONLY  non-respondents  at  least  once 
(difficult  if  anonymous) 

-  Have  PM  personally  call  all  or  their  most  valuable  customers 
asking  whether  they  have  received  the  invitation  and  if  they  will 
find  the  time  to  complete  it  (depends  on  the  #  of  participants) 

•  Increase  your  response  rate  -  After  the  Survey 

•  Keep  anonymous  respondents  anonymous 

-  Send  out  thank  you  notes  to  everyone  invited  to  participate 

-  Let  participants  know  the  results  (what  you've  done  well  and 
what  needs  to  be  improved) 

-  Provide  feedback  on  the  changes  you  plan  to  make  based  on  the 
customer  satisfaction  survey  results 
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Contact  Information 

•  Sue  LaFortune,  National  Security  Agency 

—  Phone:  443-479-6635 

—  Email:  slafort@nsa.gov 

•  Kathy  Demery,  Business  Transformation 
Institute,  Inc. 

-Phone:  330-348-3400 
—  Email:  kademery@biztransform.net 


10630  Little  Patuxent  Parkway  Ste.  405 
Columbia,  MD  21044 
www.biztransform.net 
410-997-1237 
866-301-2990  (fax) 
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Just  Getting  Started  With 
CMMI 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

Mary  Beth  Chrissis 
November  15,  2011 


-  Software  Engineering  Institute 


Carnegie  Mellon 


©2011  Carnegie  Mellon  University 


Overview 

Change  is  difficult! 

A  Harvard  Business  Review  study  finds  that  2/3  of 
change  initiatives  fail  to  achieve  successful  results. 


Fundamentals  of  change 
management  are 
like  learning  your  ABCs. 


Software  Engineering  Institute  CarnegieMellon 


Just  Getting  Started  With  CMMI 
Chrissis,  November  2011 

©2011  Carnegie  Mellon  University 
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Action-Oriented 


Results  demonstrate 
execution  and 
commitment. 

Keep  the  momentum  up. 

2-3  weeks  without 
visible  activity  causes  the 
effort  to  flounder. 


Software  Engineering  Institute  CarnegieMellon 


Just  Getting  Started  With  CMMI 
Chrissis,  November  2011 
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Budget  Appropriately 


Process  improvement/change  management  is  a 
project. 


•  plan 

•  budget 

•  resources 
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Communicate  At  All  Levels 


Others  won’t  know  unless  you  tell  them. 

Yell  out  successes. 

Acknowledge  mistakes. 

Provide  a  visible  place  for 
everyone  to  be  informed. 


Software  Engineering  Institute  CarnegieMellon 


Just  Getting  Started  With  CMMI 
Chrissis,  November  2011 
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Define  the 


Objectives 


Clearly  define  your 
objectives. 


Make  them  available 
to  the  organization. 

Connect  them  to  real 
problems  and  your 
organization’s 
business. 


This  change  effort  is  going  to 
improve  our  commitment  process. 


I r\oi  q 
fronts 


P eople  in  your  organization  fcnow  if 
they  are  relevant. 
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Encourage  Everyone  To 
Get  Involved 


Senior  Management 
Mid-Level  Management 
Supervisors 


Team  Leaders 

Teams 

Staff 


Administration 
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First,  Pick  Low  Hanging  Fruit 


The  first  change  should  be  a 
problem  that  you  believe  you  can 
successfully  fix  and  deploy 
throughout  most  of  the 
organization. 

Targets  or  goals  which  are  easily 
achievable  and  which  do  not 
require  a  lot  of  effort.  (Urban 
Dictionary) 

Pick  a  strawberry;  not  a  papaya! 


^=r  Software  Engineering  Institute  CarnegieMellon 


Just  Getting  Started  With  CMMI 
Chrissis,  November  2011 

©2011  Carnegie  Mellon  University 


Get  Management  Involved 


Cannot  be  optional  for  management. 

•  Executive  management  must  not  only 
sponsor  but  buy-in. 

•  They  must  lead  or  get  out  of  the  way. 

•  The  new  process  will  have  to  stand  on  its 
own  (aka  culture),  but  every  new  change 
needs  support  and  nurture. 

Change  efforts  must  be  coordinated. 
If  not,  employees  become  confused 
and  frustrated  (and  hence  angry) 
because  they  are  being  pulled  in 
conflicting  directions. 
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Hard  Work 


Expect  the  unexpected. 


•  Everyone  expects  that  after  the 
kick-off  meeting,  everything  goes 
smoothly. 


•  There  will  be  bumps  along  the 
way. 

Make  sure  that  you  plan  for 
the  transition  and  adoption. 

The  work  isn’t  over  until  the 
change  is  no  longer  a 
“difference”  but  is  part  of  the 
culture. 
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D  Ignore  Those  Not  Ready  to 
Change 

Most  employees  are  quite 
satisfied  with  the  status  quo- 
-20-60-20. 

Focus  on  the  60%  in  the 
middle--not  on  the  20%  that 
will  never  buy-in. 

They  will  get  on  the  band 
wagon  later  or  leave. 


IGNORE  THEM 

They  are  all  jealous 
that  I'm  not  their  rrtotfter. 


VERY  DEMOTIVATI0NAI .  .com 
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Jingle,  Jingle,  Jingle 


Keep  it  catchy! 

•  communication 

•  project  name 

•  logo 

•  people 

Developing  clear  and  catchy 
communications  that 
summarize  the  behavior 
change  which  enable  people 
to  remember  the  new 
behaviors. 


Software  Engineering  Institute  CarnegieMellon 


Just  Getting  Started  With  CMMI 
Chrissis,  November  2011 

©2011  Carnegie  Mellon  University 


Keep  It  Simple 


Remember  both  you  and  your  organization  are 
learning  about  change  management. 


Don’t  bite  off  too  much. 


Create  success  early. 


Try  to  avoid 


•  confusion 

•  lack  of  clarity 

•  lack  of  plan 

•  poor  execution 


-  Software  Engineering  Institute 


Simplicity 
is  the  ultimate 
sophistication. 


-Leon&Rdo 
dt\  Vinci 
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Leadership 


Provide  sponsorship  from  the  top. 

Identify  who  is  involved  and  who  is  responsible. 


Get  participation  from  those  affected. 


Others  will  look  for 

•  strength 

•  support 

•  direction 
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effective 


O' 

s 

‘c  ^ 

failures  rm.  lfi  Dqmmrtment  \ 3 

success^  work  Jo  | 
take  Warren 


group  CnnsGtefli 
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Measurement 


Managers  tend  to  view  events  as  successful 
without  knowing  why — they  have  no  measurements 
or  clear  expectations  about  what  the  change  will 
produce.  Staff  see  the  shortcomings  and  fewer 
advances.  It's  vital  for  the  group  to  know:  How  will 
we  know  that  we  have  gotten  to  success? 


Establish  measurement  systems  a 


the  desired  changes  and  report  the 
results  frequently. 


Never  Give 


Change  is  continuous. 


Listen  to  the  organization. 


Trust  your  instincts. 
Persevere! 
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Openness  to  Change 


Expressing  the  reasons  for  change  honestly  and 
directly  will  help  people  be  open  to  change. 


Transparency 
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p 


Plan 


Change  management  and 
process  improvement 
should  be  run  like  a  project. 


THEN  ACT 
ACCORDING  TO 
PLAN! 


Don't  fatigue  people  with 
constant  small  changes. 


Choose  big  impact  changes 
that  are  important. 
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Business 

Management 


Formal  plan 
document 


Summary 

memo 


Quickly  Respond  to  Issues 


Everyone  is  watching. 

•  supporters 

•  those  who  hope  you  fail 

Ask  for  help  when  needed! 
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Rewards  and  Resistance 


Motivate  people  to  change. 


Tangible 

•  bonuses 

•  promotion 
Intangible 

•  camaraderie 

•  sense  of  shared  destiny 
Anticipate  and  deal  with  objections  and  resistance. 

•  If  they  aren’t  addressed,  people  will  assume  they  are  true. 

•  Stay  flexible.  Be  willing  to  modify  the  process  based  on 
people’s  actions  and  events. 
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B  Survival  of  the  Fittest 

The  basis  of  survival  of  the 
fittest. 

•Adaptability 

•  Flexibility 

•  Resiliency 

The  fittest,  usually  not  the 
masses. 

Align  with  the  true 
motivations  of  your  people. 
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Think  About  What  You  Can 
Change 


Lines  of  authority  and 
control  must  be  respected; 
you  cannot  directly  change 
what  you  do  not  control. 


Example  of  a  Wide  Span  of  Control 


You  can  influence  those  in 
control,  but  you  cannot  force 
them. 


1 

r  i 

r  i 

r  i 

r  i 

r 

Employee 

Employee 

Employee 

Employee 

Employee 

Each  employee  holding  a  position  of  authority  is  responsible  for  at  least  five 
others  —  i.e.  the  span  of  control  is  at  least  5 
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Urgency  of  Need 


Set  the  stage  by  creating 
urgency  and  why  the 
change  is  important  -- 
“unfreezing.” 


Urgency  does  not  equal 
fear.  Fear  hurts.  Urgency 
helps. 


did?  to  LOOK  INSIDE! 


JOHN  P. 

KOTTER 


urgency 
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V 


Vision 


Focus  on  the  short-term 


•  3-6  months 

With  your  objectives  on 
the  near-term 

•  1  year 

But  don’t  forget  about  the 
long-  term 


6UfJt>  CArt? 


OH  ALL- KNOWING  RJdUS 

gkoo?,  uil  Ywe  uwricT 
toiJsonet  wccds,  so  that 

vJE  CAri  C  (E  EATC 
IHHOVATiOUS  FO&  Y00. 


ky  Tap^  t 


wtf  PtoMOTloMS 


YfAH,  nA  KE 
£l/€mHJ(06 


£*s  h  a  u  *  u£.  (o  Mi 


•  3  year 

•  5  year 
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What’s  In  It  For  Me? 


W 


Relate  the  change  to  what  people  in  the 
organization  want. 


It  is  a  trust  thing 


/  stuff' _ 

*  ABOUT  MG',  .  . 


THE  UNIVERSE 


o 
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eXtra,  eXtra,  eXtra 


You  will  need  help 

•  Communication 

•  Money 

•  People 
•Tools 
•Training 


Do  not  assume  that  the  level  of  enthusiasm  will 
continue,  think  about  how  you  will  sustain  that 
enthusiasm  during  the  long  road  ahead. 
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You 


It’s  not  about  you;  it’s  about  them. 

The  organization  and  individuals  must  be  able  and 
willing  to  learn  and  take  responsibility. 

Change  for  the  good  of  the  organization  and  your 
customers  first,  change  for  profit  only  second,  and 
change  for  yourself  last. 

Acknowledge  and  allow  people  to  go  through  the 
stages  of  change. 

•  They  will  anyway,  whether  or  not  you  accept  it. 


•  Expecting  it  allows  you  to  better  cope  with  it,  and  not 
overreact  to  early  denial  or  anger. 
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Zeal  But  Not  a  Zealot 


Everyone’s  buy-in  is 
important  to  change. 

Passion  and  ownership  are 
key. 


A  little  luck  or  magic  doesn’t  hurt  along  the  way! 
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Conclusion 


Basically. ..if  you 

•  define  the  objectives 

•  listen  to  your  people 

•  create  a  culture  of  change 

•  communicate  at  all  levels 

•  reward  success 

change  can  be  successful 
and  as  simple  as  your 
ABC’s. 
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Contact  Information 


Mary  Beth  Chrissis 

Sr.  Member  of  the  Technical  Staff 
SEPM  Program 
Telephone:  +1  412-268-5757 
Email:  mb@sei.cmu.edu 

Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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Background 


NORTHROP  GRUMMAN 


•  Successful  change  requires  the  right  combination  of  strategy, 
structure,  and  support 

•  Your  chances  of  success  depend  on  your  current  culture,  the 
desired  end  state,  the  resources  available,  the  past  response  to 
change ,  and  your  ability  to  recognize  and  address  resistance 

•  This  presentation  will  provide  practical  approaches,  tools,  and 
techniques  for  overcoming  resistance  in  your  organization 


This  presentation  reproduces  the  "IDEAL  Model  Graphic"  copyright  1997-2009  by  Carnegie  Mellon  University,  with  special  permission  from  its  Software 
Engineering  Institute. 
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The  IDEALSM  Model 


NORTHROP  GRUMMAN 


Learning 
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Source:  "IDEAL:  A  Users  Guide  for  Software  Process  Improvement" 
Robert  McFeeley,  CMU/SEI-96-HB-001,  Feb  1996,  used  with  permission 
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The  Non-IDEAL  Model 


NORTHROP  GRUMMAN 
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Management 
sets  a  goal  of 
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x  by  date  Y” 
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Initiating 
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The  projects  listen  politely 
(perhaps)  to  the  change 
agent’s  plans  and 
schedules,  but  either  ignore 
the  requests  for  action  or 
provide  a  minimal  response 
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Topics 


NORTHROP  GRUMMAN 


•  Necessary  ingredients  for  change 

-  Why  people  resist  change 

-  Effective  strategies  for  addressing  resistance 

•  Keys  to  leading  the  change 

-  Management  support 

-  Influence  without  authority 
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Maslow's  Hierarchy  of  Needs 


vSctuaifzatioi 


Ego  (Esteem) 


Social  (Belonging) 


Safety/ Security 


Physiological 


NORTHROP  GRUMMAN 


Opportunities  for  innovation  and 
creativity,  learning  and  creating 

Recognition  from  others,  prestige 
and  status 

Being  part  of  a  group,  identification 
with  a  team 

Economic  security,  freedom  from 
threats 

Physical  survival  needs:  food,  water, 
shelter,  etc. 
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Why  Do  People  Resist  Change? 


NORTHROP  GRUMMAN 


I  want  to  stay  where  I  am  because... 

...my  needs  are  already  met  here 
...I  have  invested  heavily  here 
...I  am  in  the  middle  of  something  important 

I  do  not  want  to  change  because... 

...the  destination  looks  worse  than  where  I  am  now 

...there  is  nothing  to  attract  me  forwards 

...I  do  not  know  which  way  to  move 

...the  journey  there  looks  painful 

...the  destination  or  journey  is  somehow  bad  or  wrong 

...I  do  not  trust  those  who  are  asking  me  to  change 

I  am  not  going  to  change  because... 

...I  am  able  to  ignore  the  change 

...I  have  the  power  to  obstruct  the  change 
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Why  Do  People  Resist  Change? 
Perceived  Loss  of  Personal  Power 


NORTHROP  GRUMMAN 


8 


©Northrop  Grumman  Systems  Corporation.  All  rights  reserved. 


Emotional  Response 


Reaction  to  Change  Perceived  as  Negative: 
Kubler-Ross  Grief  Cycle 


NORTHROP  GRUMMAN 


Passive- 


Anger 


Immobilisation  Depression 


Immobilization:  Initial  paralysis  at  hearing  the  bad  news 

Denial:  Trying  to  avoid  the  change 

Anger:  Frustration,  outpouring  of  bottled-up  emotion 

Bargaining:  Seeking  for  a  way  out 

Depression:  Final  realization  of  the  inevitable 

Testing:  Seeking  realistic  solutions 

Acceptance:  Finally  finding  the  way  forward 
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Mood 


Reaction  to  Change  Perceived  as  Positive 


NORTHROP  GRUMMAN 


Positive 


Neg  alive 
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Willingness  to  Change 


NORTHROP  GRUMMAN 


•  Early  adopters  are  motivated  by  perceived  benefits 

•  Late  adopters  are  motivated  by  avoiding  pain 

ni« 

cfl  asm 


and  parturmanre  and  cDiivEtrikGnou 
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Source:  Geoffrey  A.  Moore,  Crossing  the  Chasm,  1999,  used  with  permission 
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Topics 


NORTHROP  GRUMMAN 


•  Necessary  ingredients  for  change 

-  Why  people  resist  change 

-  Effective  strategies  for  addressing  resistance 

•  Keys  to  leading  the  change 

-  Management  support 

-  Influence  without  authority 
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Different  Practitioners  Need  Different 
Arguments 


NORTHROP  GRUMMAN 


•  (I,  EA)  Explain  how  the  practice  adds 
value 

-  Identify  a  situation  in  which  this  value 
would  be  realized 

•  (EM)  Where  possible,  provide  data 

-  It  is  difficult  to  find  perfect  data,  which 
"proves"  the  value 

-  Even  perfect  data  may  not  convince  a 
skeptic 

•  (EM)  Identify  others  who  have  seen 
the  (qualitative)  value  of  this  practice 


(LM)  Couch  the  value  in  ways 
that  appeal  to  the  practitioner: 

-  Reduced  risk,  addressing  a 
customer  issue,  reduced 
workload,  etc. 

(LM)  Show  ways  that  the 
practice  can  be  done  easily 

(LM)  Suggest  the  practitioner  try 
it  and  see 
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Communicate  the  Key  Messages 


NORTHROP  GRUMMAN 


•  The  change  is  driven  by  proven,  industry  best- practices 

-  Adoption  is  about  learning  how  to  apply  these  practices 
to  our  work 

-  The  practices  may  feel  awkward  and  have  limited  value 
until  we  learn  them 

-  It's  OK  to  make  mistakes  -  we  will  get  better  over  time 

•  I  mprovement  involves  short-term  investment  for  long-term  gain 

-  Improving  is  essential  to  meeting  our  business  goals 

•  These  improvements  are  an  enabler  (not  a  guarantee)  of  success 

-  Other  aspects  (people,  technology,  customer  relationship,  etc.)  are  equally 
important 

•  When  the  entire  organization  is  improves,  everyone's  job  becomes 
easier 

•  Continuous  improvement  is  a  way  of  life 
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Address  Fear  of  Failure 


NORTHROP  GRUMMAN 


•  The  risk  of  change  may  be  seen  as  greater 
than  the  risk  of  standing  still 

-  Making  a  change  requires  a  leap  of  faith 

•  The  perceived  loss  of  personal  power 

-  I'm  seen  as  competent  now,  but  in  a  new  culture... 

Effective  Strategies 

•  Clearly  describe  why  the  situation  favors  change 

-  Business  goals,  WIIFM 

•  Make  it  clear  initial  mistakes  are  expected  and  will  be  tolerated 

-  Create  forums  for  asking  and  answering  questions 

•  Show  people  how  they  can  be  effective  in  the  changed 
environment 
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Encourage  and  Support 


NORTHROP  GRUMMAN 


•  Practitioners  may  feel  they  don't  have  time 
to  leam  new  ideas 

•  Practitioners  may  need  role  models 

-  Most  change  agents  don't  need  role  models, 
because  they  easily  imagine  new  situations 

Effective  Strategies 

•  Ensure  adequate  resources  during  the  learning  curve 

•  Search  out  and  publicize  good  examples  and  successes 

-  Set  up  pilot  programs  that  model  the  change 

•  Encourage  the  next  step  in  the  change  process 

•  Ensure  management  takes  accountability  for  action 

-  Must  change  short  term  priorities  to  achieve  long  term  results 
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Ensure  Accountability 


NORTHROP  GRUMMAN 


•  Adopting  and  sustaining  improvements 
is  about  each  practitioner  learning  and 
performing  the  new  behaviors 

•  The  role  of  management  in  cultural  change 
is  to  hold  people  accountable  for  the  new 
behaviors  and  conduct 


Effective  Strategies 

•  Change  agents  can  enable  management  by: 

-  Helping  them  have  a  clear  vision  of  the  new  culture 

-  Identifying  inappropriate  behavior 

-  Providing  tangible,  objective  measures  of  adoption/sustainment 
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Help  Them  Accept  Change 


NORTHROP  GRUMMAN 


•  People  may  fear  hidden  agendas 

-  Late  Adopters  often  look  for  messages 
in  how  resistance  is  handled 

Effective  Strategies 

•  Set  up  mechanisms  for  obtaining  feedback 

-  Some  will  prompt  genuine  improvements 

-  Some  will  be  based  more  on  fear  and  anger  than  substance 

•  Be  honest  about  setbacks  and  negative  impacts 

•  Management  must  enforce  change  in  the  face  of  objections 

-  Consensus  will  almost  never  be  reached 

-  Communicate  that  objections  and  uncertainty  does  not  eliminate  the 
need  for  change  -  "The  dogs  may  bark,  but  the  caravan  goes  on." 
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Management  Support 


NORTHROP  GRUMMAN 


Management  must: 

•  Understand  the  key  messages 

•  Be  willing  to  take  actions  to  reinforce  them 

•  Provide  resources  to  support/  sustain  process  improvement  efforts 

•  Set  expectations  that  essential  project  functions  will  be  funded 
and  processes  will  be  followed 

-  Project  planning,  estimation,  tailoring,  CM,  QA,  etc. 

•  Support  process  improvement  and  sustainment,  rather  than 
passing  appraisals 

•  Reward  mature  processes  development  and  sustainment  rather 
than  individual  heroics 

-  Tell  me  how  you  will  reward  me,  and  I'll  tell  how  I  will  behave 
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When  Faced  with  Unexpected  Resistance 


Stop 

•  The  natural  tendency  of  many  people 
is  to  respond  immediately,  with  an 
authoritarian  or  angry  response 

•  This  may  generate  sympathy  for  the  resist' 
galvanize  the  resistance,  and/  or  make  it  c 

Look 

•  Pause,  assess  the  situation,  and  diffuse  the  emotion 

•  What  is  the  person's  emotional  state? 

Listen 

•  I  s  this  a  misunderstanding  or  a  legitimate  concern? 

•  What  does  their  message  say  about  their  underlying  beliefs, 
values,  goals,  perceptions,  potential,  triggers? 
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Principles  of  Influence 


NORTHROP  GRUMMAN 


•  All  interpersonal  behavior  involves  exchange 

-  "Paying"  others  for  what  we  request;  being  paid  for  what  we  do 

-  You  have  influence,  insofar  as  you  can  give  others  what  they  need,  in 
exchange  for  what  you  need 


•  To  have  influence,  you  must: 

-  See  the  other  person  as  a  potential  ally 

-  Clarify  your  goals  &  priorities 

-  Diagnose  your  ally's  goals  &  priorities 

-  Possess  resources  to  help  your  ally 

-  Negotiate  the  exchange 
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Possible  "Currencies"  to  Exchange 


NORTHROP  GRUMMAN 


Inspiration 

•  Vision 

•  Excellence 

•  Moral/ethical 
correctness 

Task 

•  Resources 

•  Challenge/learning 

•  Assistance 

•  Organizational  support 

•  Rapid  response 

•  Information 


Position 

•  Recognition 

•  Visibility 

•  Reputation 

•  Importance 

•  Contacts 

The 

CftHBI 


Relationship 

•  Acceptance 

•  Understanding 
Personal 

•  Gratitude 

•  Self-concept 

•  Comfort 


Customers  wsnl 
technology 
and  performance 


Customers  want 
and  ponvwiteriQU 
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Barriers  to  Seeing  the  Value 


NORTHROP  GRUMMAN 


"Sometimes  you  have  to  believe  it  to  see  it." 

•  Practitioners  may  not  have  worked  in  an  environment  where  the 
practice  was  performed 

•  Practitioners  may  have  worked  in  an  environment  where  the 
practice  was  performed  poorly  or  in  a  non-value-added  manner 

•  Believing  the  practice  is  an  improvement  may  require  an  action  the 
practitioner  is  not  willing  to  take 

-  Awkwardness  of  doing  something  new 

-  Admit  they've  been  doing  it  wrong 

-  Loss  of  personal  power  when  perceived  to  be  an  expert  in  the  current 
approach 


"The  Fundamental  Value  of  Every  Single  CMMI  Practice", 
R.  Hefner,  Wednesday,  9:15-10:00am,  Mesa  Verde 
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Summary 


NORTHROP  GRUMMAN 


•  Successful  change  requires  the  right  combination  of  strategy, 
structure,  and  support 

•  Your  chances  of  success  depend  on  your  current  culture,  the 
desired  end  state,  the  resources  available,  the  past  response  to 
change,  and  your  ability  to  recognize  and  address  resistance 
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The  CMMI  Models 


The  CMMI  Product  Suite  currently  has  three  models  relevant  to 
improvement  in  a  particular  area  of  interest. 

Development  (CMMI-DEV) 

•  build  stuff 

•  tangible,  storable  products  made  to  specification  in  a  lifecycle 

Acquisition  (CMMI-ACQ) 

•  buy  stuff 

•  specify,  solicit,  select,  contract,  procure,  accept,  transition  to  consumer 

Services  (CMMI-SVC) 

•  do  stuff 

•  intangible,  non-storable  products  delivered  via  a  service  system  based  on  explicit 
implicit  service  requests 
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RMM  &  CMMI  in  the  life  cycle 


own 


Design 


Develop 


Acquire 


CMMI-DEV 


CMMI-ACQ 


CMMI-SVC 


DEVELOPMENT 


Deploy  ^  Operate  J  Decommision 


CERT-RMM 


OPERATION 
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Relationships  Among  CMMI  Models 


Shared  PA  (SAM) 


Development-specific  PAs 


Service  “addition”  PA  (SSD) 
Service-specific  PAs 


Core  PAs 

Include  model-specific 
informative  material 


Acquisition-specific  PAs 
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What  about  Software? 


“CEOs  don’t  buy  software 
anymore... they  buy  service 
level  agreements” 

-  George  Fischer,  EVP  and  Group  Executive  for  CA  Technologies,  Speaking  at  NASSCOM  and  SEPG  AP 


-  Software  Engineering  Institute 
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SSD  “ 


”  CMMI-DEV  Engineering  PAs 
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SSD  vs.  CMMI-DEV  Engineering  PAs  i  of  4 


In  SSD  (SVC) 

In  Engineering  (DEV) 

SGI  Stakeholder  needs, 
expectations,  constraints,  and 
interfaces  are  collected, 
analyzed,  and  transformed  into 
validated  service  system 
requirements. 

RD  -  Requirements 

Development 

SP1.1  Collect  and  transform 
stakeholder  needs, 
expectations,  constraints,  and 
interfaces  into  prioritized 
stakeholder  requirements. 

RD  SG  1  Stakeholder  needs, 
expectations,  constraints,  and 
interfaces  are  collected  and 
translated  into  customer 
requirements. 

SP  1.1  Elicit  Needs 

SP  1.2  Transform  Stakeholder  Needs  into  Customer 
Requirements 

SP1.2  Refine  and  elaborate 
stakeholder  requirements  to 
develop  service  system 
requirements. 

RD  SG  2  Customer  requirements 
are  refined  and  elaborated  to 
develop  product  and  product 
component  requirements. 

SP  2.1  Establish  Product  and  Product  Component 
Requirements 

SP  2.2  Allocate  Product  Component  Requirements 

SP  2.3  Identify  Interface  Requirements 

SP1 .3  Analyze  and  validate 
requirements,  and  define 
required  service  system 
functionality  and  quality 
attributes. 

RD  SG  3  The  requirements  are 
analyzed  and  validated. 

SP  3.1  Establish  Operational  Concepts  and  Scenarios 
SP  3.2  Establish  a  Definition  of  Required  Functionality 
and  Quality  Attributes 

SP  3.3  Analyze  Requirements 

SP  3.4  Analyze  Requirements  to  Achieve  Balance 

SP  3.5  Validate  Requirements 

Software  Engineering  Institute  Carnegie  Mellon  November  2011 
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SSD  vs.  CMMI-DEV  Engineering  PAs  2  of  4 


In  SSD  (SVC) 

In  Engineering  (DEV) 

SG  2  Service  system 
components  are  selected, 
designed,  implemented,  and 
integrated. 

TS  -  Technical  Solution 

PI  -  Product  Integration 

SP  2.1  Select  service  system 
solutions  from  alternative 
solutions. 

TS  SGI  Product  or  product 
component  solutions  are  selected 
from  alternative  solutions. 

SP  1.1  Develop  Alternative  Solutions  and  Selection 
Criteria 

SP  1.2  Select  Product  Component  Solutions 

SP  2.2  Develop  designs  for  the 
service  system  and  service 
system  components. 

TS  SG  2  Product  or  product 
component  designs  are 
developed. 

SP  2.1  Design  the  Product  or  Product  Component 

SP  2.2  Establish  a  Technical  Data  Package 

SP  2.3  Design  Interfaces  Using  Criteria 

SP  2.4  Perform  Make,  Buy,  or  Reuse  Analyses 

SP  2.3  Manage  internal  and 
external  interface  definitions, 
designs,  and  changes  for 
service  systems. 

PI  SG  1  Preparation  for  product 
integration  is  conducted. 

PI  SG  2  The  product-component 
interfaces,  both  internal  and 
external,  are  compatible. 

SP  1.1  Establish  an  Integration  Strategy 

SP  1.2  Establish  the  Product  Integration 

Environment 

SP  1.3  Establish  Product  Integration  Procedures  and 
Criteria 

SP  2.1  Review  Interface  Descriptions  for 
Completeness 

SP  2.2  Manage  Interfaces 

=-  Software  Engineering  Institute  Carnegie  Mellon  November  2011 
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SSD  vs.  CMMI-DEV  Engineering  PAs  3  of  4 


In  SSD  (SVC) 

In  Engineering  (DEV) 

SP  2.4  Implement  the  service 
system  design. 

TS  SG  3  Product  components, 
and  associated  support 
documentation,  are  implemented 
from  their  designs. 

SP  3.1  Implement  the  Design 

SP  3.2  Develop  Product  Support  Documentation 

SP  2.5  Assemble  and  integrate 
implemented  service  system 
components  into  a  verifiable 
service  system. 

PI  SG  3  Verified  product 
components  are  assembled  and 
the  integrated,  verified,  and 
validated  product  is  delivered. 

SP  3.1  Confirm  Readiness  of  Product  Components 
for  Integration 

SP  3.2  Assemble  Product  Components 

SP  3.3  Evaluate  Assembled  Product  Components 

SP  3.4  Package  and  Deliver  the  Product  or  Product 
Component 
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SSD  vs.  CMMI-DEV  Engineering  PAs  4of4 


In  SSD  (SVC) 

In  Engineering  (DEV) 

SG  3  Selected  service  system 
components  and  services  are 
verified  and  validated  to 
ensure  correct  service 
delivery. 

VER  -  Verification 

VAL  -  Validation 

VER  SP  1.1  Select  Work  Products  for  Verification 

VER  SP  1.2  Establish  the  Verification  Environment 
VER  SP  1.3  Establish  Verification  Procedures  and 
Criteria 

VAL  SP  1.1  Select  Products  for  Validation 

VAL  SP  1.2  Establish  the  Validation  Environment 

VAL  SP  1.3  Establish  Validation  Procedures  and 
Criteria 

SP  3.1  Establish  and  maintain 
an  approach  and  an 
environment  for  verification  and 
validation. 

VER  SG  1  Preparation  for 
verification  is  conducted. 

VAL  SG  1  Prepare  for  validation 
is  conducted. 

SP  3.2  Perform  peer  reviews  on 
selected  service  system 
components. 

VER  SG  2  Peer  reviews  are 
performed  on  selected  work 
products. 

VER  SP  2.1  Prepare  for  Peer  Reviews 

VER  SP  2.2  Conduct  Peer  Reviews 

VER  SP  2.3  Analyze  Peer  Review  Data 

SP  3.3  Verify  selected  service 
system  components  against 
their  specified  requirements. 

VER  SG  3  Selected  work 
products  are  verified  against  their 
specified  requirements. 

VER  SP  3.1  Perform  Verification 

VER  SP  3.2  Analyze  Verification  Results 

SP  3.4  Validate  the  service 
system  to  ensure  that  it  is 
suitable  for  use  in  the  intended 
delivery  environment  and  meets 
stakeholder  expectations. 

VAL  SG  2  The  product  or  product 
components  are  validated  to 
ensure  they  are  suitable  for  use 
in  their  intended  operating 
environment. 

VAL  SP  2.1  Perform  Validation 

VAL  SP  2.2  Analyze  Validation  Results 
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Different  Names  and  Abbreviations 
of  Some  Core  PAs 


SVC 

Work  Planning  (WP) 

Work  Monitoring  and  Control 
(WPM) 

Integrated  Work  Management 
(IWM) 

Quantitative  Work  Management 
(QWM) 


DEV 

Project  Planning  (PP) 

Project  Monitoring  and  Control 
(PMC) 

Integrated  Project  Management 
(IPM) 

Quantitative  Project  Management 
(QPM) 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  SVC  &  DEV 
November  2011 

©2011  Carnegie  Mellon  University 


Differences  in  PAs  and  Categories 


CMMI-SVC  PAs  by  Category 

Process  Management 

Project  and  Work  Management 

Support 


Service  Establishment 
and  Delivery 


CMMI-DEV  PAs  by  Category 

Process  Management 

Project  Management 


Support 


ijii 


Engineering 
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One  Example  Group:  Urban  Transit  Authority 
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Another  Group:  Suppliers  of  Trains,  Control 
Systems,  Entry  Gates,  etc. 
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Transit  Authority  vs.  Equipment  Suppliers: 
What  Do  They  Have  in  Common? 


Both  are  trying  to  deliver  value  to  a  customer. 
Both  need  to  understand  the  customer’s  needs. 
Both  need  to  plan  a  solution  to  those  needs. 
Both  need  to  validate  and  deliver  that  solution. 
Both  need  good  results  to  stay  in  business. 
Other  ideas? 
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Transit  Authority  vs.  Equipment  Suppliers: 
How  Do  They  Differ? 


Transit  Authority 

•  delivered  solution  is  intangible, 
non-storable 

•  ongoing  relationship  based  on  a 
service  agreement 

•  services  often  simultaneously 
produced  and  consumed 

•  more  time  spent  on  delivery 


Equipment  Suppliers 

•  delivered  solution  is  a  tangible, 
physical 

•  fixed-term  relationship  based  on  a 
delivery  contract 

•  delivery  of  product  generally  takes 
place  after  development  (and 
maybe  after  manufacturing) 

•  more  time  spent  on  development 


Goods  _ Develop _ l  Deliver 

Service  Develop  | _ Deliver _ 
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Differences  in  Process  Improvement  Between 
Services  and  Development 


Services 

•  Changes  are  made  to  the  service 
system,  which  immediately  affects 
delivery  of  services  to  customer. 

•  Performance  of  the  service  system 
is  inseparable  from  quality  of 
service. 

•  Feedback  from  service  users  to 
providers  is  typically  direct  and 
rapid. 

•  Repeated  service  delivery  provides 
numerous  and  frequent 
opportunities  to  measure. 


Development 

•  Changes  are  made  to  development 
methods  and  tools,  but  impacts 
may  not  be  visible  to  the  customers 
until  later  product  deliveries. 

•  The  distinction  between 
development  process  performance 
and  product  performance  is  clear. 

•  Feedback  from  product  end  users 
to  developers  is  often  indirect  and 
slow. 

•  Longer  development  cycles  provide 
more  limited  and  less  frequent 
opportunities  to  measure. 
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How  Might  Services  PAs  Help  Development? 


Although  they  are  not  included  in  the  CMMI-DEV  model,  each  of  the 
following  CMMI-SVC  process  areas  could  be  used  by  a  development 
group: 

•  Capacity  and  Availability  Management 

•  Service  Continuity 

•  Incident  Resolution  and  Prevention 

•  Service  System  Transition 

•  Strategic  Service  Management 

We  have  examples  of  high  maturity  development  organizations  doing 
system  of  systems  engineering  who  are  finding  the  CMMI-SVC  PAs  add 
new  capability. 
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A  Look  at  CMMI-SVC 


Shared  PA 
(SAM) 


( 

)  CMMI-SVC 

t 

a 

Services-specific  PAs 
*CMMI-SVC  addition 


Core  PAs 

Include  service-specific 
informative  material 


CMMI-ACQ 


/ 


Capacity  and 

Availability 
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Another  look  at  STSM 

Strategic  Service  Management  (STSM): 

STSM  is  about  portfolio  management  or  deciding  what  services  you  should  be 
providing,  making  them  standard,  and  letting  people  know  about  them. 

Why  do  the  practices  in  STSM?  Because  you  have  standard  services, 
developing  new  services  is  faster  and  cheaper.  You  can  increase  business 
capture  and  market  share.  You  and  your  customers  agree  about  what  you  have 
to  offer. 

In  an  engineering  services  context: 

•  For  a  customer-intensive  service  like  this,  it’s  tempting  to  do  anything  asked. 
Using  the  practices  of  this  PA  to  get  clear  about  what  offerings  are  in  your 
best  interest  along  with  your  clients’  interests  can  give  you  business 
advantage.  Retire  and  improve  services  as  well  as  adding  to  your  catalogue. 

•  Describing  your  offerings  appropriately  for  your  customers  may  be  obvious. 
Less  obvious  are  the  benefits  from  being  disciplined  inside  your  engineering 
service  company  about  what  products  you  offer  and  why — good  internal 
descriptions  can  help  to  foster  that  discipline. 
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You  run  a  service  group  inside  a  larger  company. 

You  are  thinking  of  applying  the  practices  of  STSM  in  your  organization. 

Do  you  have  to  wait  for  “corporate”  to  build  their  service  catalog  before 
you  build  one? 


1 

No 

2 

Yes 

5 

Not  Sure 
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Another  look  at  SD 

Service  Delivery  (SD): 

SD  is  about  setting  up  agreements,  taking  care  of  service  requests,  and 
operating  the  service  system. 

Why  do  the  practices  in  SD?  You  and  your  customer  have  the  same 
expectations,  your  services  are  consistent  and  cost-effective,  and  customers 
know  how  to  make  requests. 

In  an  engineering  services  context: 

•  Service  agreements  often  begin  during  the  proposal  period  and  are  refined 
after  a  win.  They  should  also  be  maintained  regularly. 

•  Thinking  of  “readiness”  of  your  service  system  as  an  ongoing  quality  attribute, 
not  something  done  just  at  the  beginning  of  an  engagement.  This  orientation 
is  one  of  the  key  differences  between  service  and  development,  and 
particularly  likely  when  your  service  is  related  to  development. 

•  Request  management  appears  to  be  one  of  the  strong  leverage  points  for 
business  results  in  this  type  of  service. 
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Another  look  at  CAM 


Capacity  and  Availability  Management  (CAM): 

CAM  is  about  making  sure  you  have  the  resources  you  need  to  deliver  services 
and  that  they  are  available  when  needed — at  an  appropriate  cost. 

Why  do  the  practices  in  CAM?  Customer  satisfaction  is  increased  because  of 
high  availability  of  service.  Your  costs  are  managed  and  the  risk  of  service 
failure  is  reduced. 

In  an  engineering  services  context: 

•  Like  other  people-intensive  services,  treating  your  people  as  the  key 
resource  that  concerns  you  for  capacity  and  availability  will  be  part  of  your 
CAM  strategy.  In  addition  to  number  and  availability  of  staff,  skill  availability  is 
also  crucial. 

•  Patterns  of  surge  and  lag  times,  and  issues  like  overwork  for  favorite  or 
productive  or  skilled  staff  deserve  attention. 
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Another  look  at  SSD 


Service  System  Development  (SSD): 

SSD  is  about  making  sure  you  have  everything  you  need  to  deliver  the  services, 
including  people,  processes,  consumables,  and  equipment. 

Why  do  the  practices  in  SSD?  You  anticipate  service  requirements  and  avoid 
costly  changes.  The  service  system  does  what  is  required  for  both  the  service 
provider  and  customer. 

In  an  engineering  services  context: 

•  You  probably  have  a  service  system  that  is  enterprise  wide,  or  specific  to  a 
product  line,  or  in  some  way  larger  than  one  engagement  or  customer.  The 
practices  in  this  PA  should  also  be  used  to  alter,  adjust,  or  tailor  for  individual 
agreements. 

•  This  PA,  and  its  wording  that  relies  heavily  on  IT  and  engineering  can  be 
difficult  or  misleading  for  services  that  are  NOT  based  in  engineering;  here’s 
one  place  in  the  model  where  engineering  services  has  an  advantage! 
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When  to  use  Service  System  Development 
(SSD) 

Engineering  PAs  in  DEV  are  recommended  for  improving  product 
development  process,  large  complex  systems,  and  those  very  familiar 
with  DEV. 

Using  SSD  may  be  preferred  by  service  provider  organizations  that  are 
new  to  CMMI — especially  those  organizations  with  simple  services. 

Even  organizations  that  use  the  CMMI-DEV  model  for  service  system 
development  may  refer  to  the  SSD  process  area  for  helpful  guidance  on 
applying  development  practices  to  service  system  parts  like  people, 
processes,  and  consumables. 

Two  places  in  the  model  to  look  for  help  for  small,  simple  services  if 
SSD  is  too  much: 

•  Goal  2  in  Service  Delivery 

•  SP  1.3  in  IWM,  Establish  the  Project’s  Work  Environment 
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SAM  is  to  CMMI-ACQ  as  is  to  CMMI-DEV 
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SD 
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CAM 
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SST 
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SSD 
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1  don’t  have  a  clue 
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Your  service  is  vehicle  maintenance. 


Though  maintenance  is  your  major  service,  occasionally  you  build  a  small 
widget  and  install  it  on  the  vehicles  you  maintain.  You  are  arguing  with  your 
process  improvement  consultant  about  which  model  content  to  apply,  partly 
because  you  have  a  SCAMPI  appraisal  coming,  and  she  wants  you  to  do  well. 
What  model  content  would  you  apply  in  your  context? 


1 

SSD  can  be  applied  to  developing  the  widget. 

2 

SSD  covers  the  service  system,  but  engineering  PAs  should  be  used 
for  developing  the  widget. 

3 

Engineering  PAs  could  be  used  for  widget  and  service  system. 

4 

What  does  it  matter?  Apply  whichever  practices  help  us  to  improve. 
Ignore  the  appraisal. 

5 

This  kind  of  question  is  what  makes  me  crazy  about  CMMI. 
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Another  look  at  SST 

Service  System  Transition  (SST): 

SST  is  about  getting  new  systems  in  place,  changing  existing  systems,  or 
retiring  obsolete  systems — all  while  making  sure  nothing  goes  terribly  wrong  with 
the  service. 

Why  do  the  practices  in  SST?  Your  service  delivery  doesn’t  degrade  when  you 
make  a  major  change.  You  minimize  customer  and  user  dissatisfaction  and 
transition  smoothly  into  and  out  of  operations. 

In  an  engineering  services  context: 

•  Smoothly  handling  changes  in  any  component:  your  people,  tools, 
consumables,  and  more  is  an  essential  element  of  superior  service. 

•  It’s  not  unusual  in  this  service  to  take  over  from  or  hand  over  to  another 
vendor.  You  also  must  end  your  service  gracefully.  All  of  these  conditions 
are  aided  by  the  practices  in  SST. 

•  For  all  service  types,  including  stakeholders  appropriately  during  transition  is 
a  capability  that  distinguishes  excellent  providers  from  their  peers. 
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Another  look  at  SCON 

Service  Continuity  (SCON): 

SCON  is  about  being  ready  to  recover  from  a  disaster  and  get  back  to  delivering 
your  service. 

Why  do  the  practices  in  SCON?  After  9/1 1  and  Katrina,  service  businesses  have 
proof  that  those  who  prepare  for  disaster  are  better  able  to  recover  and  stay  in 
business. 

In  an  engineering  services  context: 

•  It  may  be  tempting  to  think  this  business  is  relatively  resilient  to  disastrous 
disruption.  With  recent  natural  disasters,  it  may  look  different  now.  With  a 
community  disaster,  even  if  you’ve  considered  your  essential  resources  and 
dependencies,  can  you  withstand  a  protracted  delay  in  getting  back  to 
business? 

•  With  people  being  the  key  resource,  how  can  you  equip  them  to  be  resilient  to 
a  range  of  disasters  or  significant  disruptions? 
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Another  look  at  IRP 

Incident  Resolution  and  Prevention  (IRP): 

IRP  is  about  handling  what  goes  wrong — and  preventing  it  from  going  wrong 
before  it  occurs  if  you  can. 

Why  do  the  practices  in  IRP?  Services  can  continue,  even  when  something  goes 
wrong,  because  you  know  how  to  work  around  incidents.  You  address 
underlying  causes  of  incidents  so  that  you  reduce  costs  and  other  adverse 
impacts. 

In  an  engineering  services  context: 

•  Stay  clear  that  “incidents”  are  disruptions  to  your  engineering  service,  not 
bugs  or  defects  you  may  hired  to  find  or  resolve. 

•  Have  a  strategy  for  deciding  what  to  work  around  and  what  calls  for  root 
cause  analysis  and  prevention. 
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CMMI-SVC  Service  PAs  in  Plain  Language 

Capacity  and  Availability  Management  (CAM): 

making  sure  you  have  enough  of  the  resources  you  need  to  deliver  services  and  that  they  are 
available  when  needed — at  an  appropriate  cost 

Incident  Resolution  and  Prevention  (IRP): 

handling  what  goes  wrong — and  preventing  it  from  going  wrong  ahead  of  time  if  you  can 

Service  Continuity  Management  (SCON): 

being  ready  to  recover  from  a  disaster  and  get  back  to  delivering  your  service 

Service  Delivery  (SD): 

setting  up  agreements,  taking  care  of  service  requests,  and  operating  the  service  system 

Service  System  Development  (SSD): 

making  sure  you  have  everything  you  need  to  deliver  the  service,  including  people,  processes, 
consumables,  and  equipment 

Service  System  Transition  (SST): 

getting  new  systems  in  place,  changing  existing  systems,  and  retiring  obsolete  systems,  all  while 
making  sure  nothing  goes  terribly  wrong  with  service 

Strategic  Service  Management  (STSM): 

deciding  what  services  you  should  be  providing,  making  them  standard,  and  letting  people  know 
about  them 
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Core  and  Shared  PAs  in  Plain  Language  -i  of  3 

Causal  Analysis  and  Resolution  (CAR): 

getting  to  the  sources  of  important  outcomes  and  taking  effective  action  to  correct  or  repeat  them 

Configuration  Management  (CM) 

controlling  changes  to  your  crucial  work  products 

Decision  Analysis  and  Resolution  (DAR): 

using  a  formal  decision  making  process  on  the  decisions  that  matter  most  in  your  business 

Integrated  Work  Management  (IWM): 

making  the  most  of  your  participants  and  defined  processes,  even  when  it’s  complex 

Measurement  and  Analysis  (MA): 

knowing  what  to  count  and  measure  to  manage  your  service 

Organizational  Performance  Management  (OPM): 

managing  your  improvements  and  innovations  using  a  statistical  understanding  of  your  process 
performance 

Organizational  Process  Definition  (OPD): 

establishing  standard  processes  and  relaying  them  throughout  your  organization 
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Core  and  Shared  PAs  in  Plain  Language  -2  of  3 

Organizational  Process  Focus  (OPF): 

figuring  out  your  current  process  strengths  and  weaknesses,  planning  what  to  do  to  improve,  and 
putting  those  improvements  in  place 

Organizational  Process  Performance  (OPP): 

making  sure  you  understand  your  process  performance  and  how  it  affects  service  quality 

Organizational  Training  (OT): 

developing  the  skills  and  knowledge  your  people  need  to  deliver  superior  service 

Process  and  Product  Quality  Assurance  (PPQA): 

checking  to  see  that  you  are  actually  doing  things  the  way  you  say  you  will  in  your  policies, 
standards,  and  procedures 

Quantitative  Work  Management  (QWM): 

managing  service  to  quantitative  process  and  performance  objectives 

Requirements  Management  (REQM): 

keeping  clear  with  your  customers  and  other  stakeholders  about  the  service  you  provide,  and 
adjusting  when  you  find  inconsistency  or  mismatched  expectations 

Supplier  Agreement  Management  (SAM): 

getting  what  you  need  and  what  you  expect  from  suppliers  who  affect  your  service 
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Core  and  Shared  PAs  in  Plain  Language  -3  of  3 

Risk  Management  (RSKM): 

supporting  the  success  of  your  service  mission  by  anticipating  problems  and  how  you  will  handle 
them — before  they  occur 

Work  Monitoring  and  Control  (WMC): 

making  sure  what’s  supposed  to  be  happening  in  your  service  work  is  happening  and  fixing  what  isn’t 
going  as  planned 

Work  Planning  (WP): 

estimating  costs,  effort,  and  schedules;  getting  commitment  to  the  work  plan;  and  involving  the  right 
people — all  while  watching  your  risks  and  making  sure  you’ve  got  the  resources  you  think  you  need 
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CMMI-DEV  Engineering  PAs  in  Plain  Language 

Product  Integration  (PI): 

putting  together  all  the  product  components  so  that  the  overall  product  has  expected  behaviors  and 
characteristics 

Requirements  Development  (RD): 

understanding  what  stakeholders  think  they  need  and  documenting  that  understanding  for  the  people 
who  will  be  designing  solutions 

Technical  Solution  (TS): 

using  effective  engineering  to  build  solutions  that  meet  end  user  needs 

Validation  (VAL): 

making  sure  that  the  solution  actually  meets  the  needs  of  users  in  the  service  environment 

Verification  (VER): 

making  sure  that  the  solution  you  ended  up  with  meets  your  agreement  about  the  needs 
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Putting  All  the  Pieces  Together 
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What’s  the  Summary? 


CMMI-SVC  has  a  PA  that  “summarizes”  the  engineering  PAs  in  DEV, 
for  those  occasions  when  more  detailed  practice  information  is  needed. 

CMMI-SVC  and  CMMI-DEV  can  be  used  and  appraised  together. 

Development  or  engineering  tasks  can  be  treated  as  a  service,  and 
managed  with  the  practices  in  CMMI-SVC. 

Advanced  development  may  use  all  of  the  CMMI-DEV,  and  then  add 
CMMI-SVC  for  additional  practices:  SCON,  SST,  CAM. 

If  we  take  a  life  cycle  view  and  consider  total  cost  of  ownership,  may 
need  multiple  models,  a  mash  up  or  composition  from  CMMI  and  other 
models. 

Other  ideas? 
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Contact  information 


Eileen  Forrester 

Manager,  CMMI  for  Services 
SEPM 

Telephone:  +1  412-268-6377 
Email:  ecf@sei.cmu.edu 


Web 

www.sei.cmu.edu/cmmi 

www.sei.cmu.edu 
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The  Future  of  CMMI  -  Panel  NPIK 

National  Defense  Industrial  Association 

Introduction: 

•  How  well  does  CMMI  meet  the  needs  of  your 
organizations,  and  the  marketplace  in  general? 

•  What  works?  What  doesn’t?  Where  does  CMMI 
need  to  go  to  better  fit  the  needs  of  organizations 
in  the  future? 

Format: 

•  Panelist  responses  to  topic  statements 

•  Open  discussion  /  Q&A 
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Panelists 


NPIK 

National  Defense  Industrial  Association 

•  Mike  Campo  (Raytheon  Company) 

•  Brian  Gallagher  (Northrop  Grumman) 
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•  Lynn  Penn  (Lockheed  Martin) 

•  Rusty  Young  (Software  Engineering  Institute) 
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PEF  program  types 


NORTHROP  GRUMMAN 


1 .  Product  Development.  Activities  involved  in  the  transformation  of  customer  needs  to 
delivered  products  or  service  systems  required  to  deliver  services. 

2.  Product  Maintenance.  Activities  involved  in  the  adaptive,  corrective,  improvement, 
enhancement  and  sustainment  of  delivered  products 

3.  Production  /  Manufacturing.  Activities  to  repetitively  produce  products  with  no  or  slight 
variations  on  an  approved  core  design. 

4.  Staff  Augmentation.  Activities  related  to  providing  consulting  expertise  with  process 
ownership  of  program  management  only;  delivering  hours  only.  Management  of  the 
activities  is  done  by  the  customer. 

5.  Professional  Services.  Activities  related  to  providing  services  as  specified, 

including  ownership  of  essential  processes.  Unique  program  management  of  the  activities 
is  done  by  NG. 

6.  T  Managed  Services.  Activities  related  to  providing  Information  Technology  infrastructure 
services  to  organizations. 

7.  Operations.  Operations,  routine  maintenance,  and  /  or  support  to  accepted,  deployed, 
operational  systems. 

8.  Product  Line.  Activities  to  build  a  set  of  systems  or  products  sharing  a  common,  managed 
set  of  features  that  satisfy  the  specific  needs  of  a  particular  market  segment  or  mission 
and  that  are  developed  from  a  common  set  of  core  assets  in  a  prescribed  way. 
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Key  Tenets 
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•  Affordability  is  the  practice  of  ensuring  program  success 
through  the  balancing  of  system  performance  (KPPs), 
total  ownership  cost ,  and  schedule  constraints  while 
satisfying  mission  needs  in  concert  with  long-range 
investment,  and  force  structure  plans  of  the  DoD 

•  To  meet  these  principles  we  are  recommending  guidance 
for  Affordability  and  SE  Tradeoff  Analyses  that  yield 
visibility  Into  the  relationship  among  the  life-cycle  phases, 
the  KPPs  and  mission  effectiveness.  Key  issues  to  be 
resolved 

-  Quality  of  empirical  data  for  estimation 

-  Ensuring  compatibility  of  affordability  principles  with  budget  and 
competitive  procurement  policies  and  processes 

-  Life-cycle  RAA  (responsibility,  authority  &  accountability) 


NDiA  Systems  Engineering  Division 

8/9/2011 


Affordabrfity  Working  Group 
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Summary 
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•  CMMI  can  help  diverse  cultures  establish  a  common  process  framework 
to  enable  Program  Execution  success 

•  Strong,  disciplined  acquisition  and  development  processes  based  on 
CMMI  models  (ACQ,  DEV,  SVC)  is  essential  for  meeting  affordability 
challenges 


15 


©  Copyright  2011  Northrop  Grumman 


NORTHROP  GRUMMAN 


NORTHROP  GRUMMAN 


CMMI  User’s  Group  Conference,  2011 


November  15,  201 1 
Brian  P.  Gallagher 

Division  Director 
Cyber  Intelligence  Division 
Northrop  Grumman  Information  Systems 


©  Copyright  2011  Northrop  Grumman 


NGC 

Capabilities  at  a  Glance 


NORTHROP  GRUMMAN 


•  Weaponized  Platforms 

*  Resilient  Systems 


•  Weaponized  Payloads 

•  Non-Kinetic  Effects 

•  Secure  Supply  Chain 

•  Cyber  Capability 


•  Training  Programs 

*  Facility  Staffing 


*  CND  Operations  (1C,  DoD,  Fed) 

*  Title  50  &  Title  10  Operations 

*  Intelligence  Fusion  &  Analysis 

*  NCC,  Kinetic  &  Non-Kinetic  C2 

*  Next  Generation  Networks 


Aerospace 

Systems 


Electronic 

Systems 


Technical 

Services 


Integrated  Cyber  Security  Goals  and  Future  Vision 


;i 


j  Corporate 
Cross  Sector 
Initiatives 


•  Multi-lnt,  Multi-Sensor  Mission 
Assurance 


•  Project  Viceroy 

•  Project  NKE 


•  Coordinated  Non-Kinetic  and  Kinetic 
Mission  Assurance 


Committed  to  Addressing  The  Nation’s  Challenges 


2 


©  Copyright  2011  Northrop  Grumman 


Standardization:  The  Program  Execution 
Framework  (PEF)  Reference  Model 


NORTHROP  GRUMMAN 


Program  Management 


Business  Acquisition  Process 


Bd  Proposal 


Enabling 


Production 


Post  Award 

¥ 

Program 
___  Startup 

Decision  Signoff 

Development 

Requirements 

Design 

Implementation 

Integration 

Acceptance 

A 

Requirements 

IPR 

l  A  A  i 

i  Preliminary  Rinai  Design/  Ti 

Design  Implementation  Read 

(PR  Readiness  IF 

L  A 

Readiness 

iness  (PR 

R 

Production  Planning 


Material  Management 


Production  Assembly 


Production  Test 


Production  Delivery 

I - 


Production 
Readiness  IPR 


~sr~ 

Periodic 

(PR* 


Program 

Planning 

Earned  Value 
Management 

Program 
Monitoring  and 
Control 

Risk  and 
Opportunity 
Management 

Program  Supply 
Chain 

Management 

Performance 

Management 

Improvement 

Management 

Service 


Service  Delivery 


Availability  Management 


Capacity  Management 


Continuity  Management 


Product  Line  Management 

2  £ 

Product  Line  Periodic 

Readiness  IPR  (PR* 


Incident  Management 

i  A 

Readiness  (PR  (PR* 


Configuration  /  Data 
Management 

Mission  Assurance 

Process  Engineering 

Reviews  and  Assessments 

Structured  Decision  Making 

Validation 

Verification 

Work  Environment 

Reference  model  shows  all  activities  for  all  program  types 


3 


©  Copyright  2011  Northrop  Grumman 


PEF  program  types 


NORTHROP  GRUMMAN 


1 .  Product  Development.  Activities  involved  in  the  transformation  of  customer  needs  to 
delivered  products  or  service  systems  required  to  deliver  services. 
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enhancement  and  sustainment  of  delivered  products 
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Complex  Systems 


•  Complex  systems  are  already  woven  tightly 
into  our  everyday  lives 

•21st  century  systems  will  be  increasingly 
complex 

•  How  failing  to  understand  them,  their  risks, 
and  their  management  challenges  poses  a 
21st-century  hazard 
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The  Rise  of  Complexity 


•  Scale 

•  Interconnectedness 

•  Autonomy 

•  Time  criticality 

•  Security 

•  Safety 

•  Regulation 
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Complexity  Brings  Rewards  -  and 


Bad  things  happen  when  we  ... 
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...  complex  systems 

Average  person  does  not  know  or 
care  about  root  cause  of  problem 
in  system  complexity 
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Examples:  Good  ...  and  Bad 
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An  Interconnected  Society 
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Autonomous  Systems 
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Human  System  Interaction 
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Complex  Systems  at  the  SEI 


The  SEI  is  at  the  nexus  of  systems  and 
complexity: 

•  We  study  them  side-by-side 

•  For  25  years,  we’ve  been  helping  engineers 
design  and  manage  software  systems 

•  It’s  our  job  to  “ring  the  bell”  on  the  importance 
of  managing  complexity 

We  also  appreciate  risk  and  the  importance 
of  managing  it 

•  Process  improvement 

•  Risk  management  and  resilience 

•  Architectural  approaches 

•  Multi-view  models 
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Software  is  Everywhere 
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Software  is  Important 


Manufacturing 
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Software  is  Increasingly  Complex 
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Software  Connects  Us 


Wireless  Communications  Technology  Map 
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Software  is  Becoming  More  Personal 


A  Super  Smart  Grid 
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“Software  is  Eating  the  World” 


Marc  Andreessen  essay,  August  20,  2011 

•  Software  is  eating  the  world  ...  it  is  everywhere 

•  “A  trend  I've  observed,  one  that  makes  me  optimistic  about  the  future  growth 
of  the  American  and  world  economies  ..." 

•  “More  and  more  major  businesses  and  industries  are  being  run  on  software 
and  delivered  as  online  services — from  movies  to  agriculture  to  national 
defense.” 


How  to  Handle  Complexity 


Models,  Process  and  Process  Improvement 
Architecture 

Risk  Management  and  Resiliency 
Evolution  and  Disruption 
People 
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Process  Improvement,  People,  and  Models 


Process  Improvement  helps  us 


standardize,  examine  and 
improve  how  we  work 

•  Process  improvement  also  lets 
us  learn  from  others  and 
ourselves 


Models  help  us  simplify 


complex  systems  and  give  us 
insightful  views 


Process  improvement  and 
models  are  tools  for  people, 
their  masters 

•  People  design,  develop  and 
operate  systems 


Why  Use  CMMI? 


Makes  you  more  competitive 
Improves  your  quality 
Links  you  with  a  global  market 
Scales  to  your  needs 
Helps  you  prioritize 
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The  Key  Ideas  of  CMMI 


Elements: 

•  know  your  customer 

•  know  your  company 

•  focus  on  quality 

•  commit  to  continuous  improvement 

•  make  decisions  based  on  data 


Benefits: 

•  deliver  quality  products 

•  fewer  delivered  defects 

•  less  rework 

•  meet  schedules 

•  reduce  costs  profits 

•  quicker,  more  agile  cycles 

•  more  business 
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TSP  -  Software  Engineering  Best 


Small  App. 

I 


Start 

I 


Size 


Medium  App. 

I 


Scalable... small  to  large  projects 
Supports  broad  portfolio  of  work 
Best  standard  practice 

Large  App. 

I 


Rank 

Method 

1 

Agile 

2 

TSP/PSP 

3 

Waterfall 

4 

CMMI  ML2 

|  Rank | 

Method 

i 

TSP/PSP 

2 

Agile 

3 

CMMI  ML3 

4 

RUP 

|  Rank  | 

Method 

N  i 

TSP/PSP 

2 

CMMI  3,4,  5 

3 

RUP 

4 

Hybrid 

Best  development  practices  by  size  of  application1 


1.  Software  Engineering  Best 
Practices,  C.  Jones,  2010 


Software 
Engineering 
Best  Practices 
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Software  Engineering  Issues  in  R&D 


U.S.  SOFTWARE  PERFORMANCE  LEVELS 


PROJECT  TECHNICAL  SOFTWARE 

MANAGEMENT _ STAFFS _ USERS 


Sizing 

Fair 

Requirements 

Fair 

Requirements 

Poor 

Estimating 

Poor 

Design 

Good 

Schedule  Demands 

Poor 

Planning 

Fair 

Coding 

Good 

Reviews 

Fair 

Tracking 

Poor 

Reviews 

Fair 

Acceptance  Test 

Fair 

Measuring 

Poor 

Testing 

Good 

Usage 

Good 

Overall 

Poor 

Good 

Fair 

Conclusion:  U.  S.  technical  skills  are  better  than  U.  S.  management  skills. 

Project  management  and  quality  are  frequent  problem  areas. 
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From  Team  Software  Process  (TSP)  in  Context,  by  Capers  Jones 
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Accelerated  Improvement  Method  (AIM) 


Integrates  and  Leverages  SEI  Technologies 


CMMI 
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AIM  -  Predictably  Improves  Software  Engineering 
Practice 


Summary  of  field  trial  results 

•  rapid  deployment  -  15  months 

•  tactical  project-focused  approach 

•  supports  implementation  in¬ 
progress  on  critical  software 
engineering  projects 

•  predictable  improvement  timeline 
and  cost 

•  faster,  higher  ROI  addresses  “no¬ 
time”  and  “can’t  afford”  barriers  to 
change 

•  same  TSP  performance  benefits 


AIM  Results 

CGI  Results 

Typical 

Range 

CMMI-DEV  ML1  to 
ML3  (months) 

15 

36  to  48 

Schedule  Variance 

<10% 

25%  to  100% 

Effort  Variance 

7% 

25%  to  100% 

Productivity 

up  35% 

Quality  (defects) 

down  50% 
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Architecture  is  Important 


The  quality  and  longevity  of  a  software-reliant  system  is  largely 
determined  by  its  architecture. 


In  recent  studies  by  OSD,  the  National  Research  Council,  NASA,  and 
the  NDIA,  architectural  issues  are  identified  as  a  systemic  cause  of 
software  problems  in  DoD  systems. 
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Why  is  Software  Architecture  Important? 


Represents  earliest 
design  decisions 


•  hardest  to  change 

•  most  critical  to  get  right 

•  communication  vehicle 
among  stakeholders 


First  design  artifact 
addressing 


performance 

modifiability 

reliability 

security 


Key  to  systematic  reuse 


•  transferable, 
reusable  abstraction 


Key  to  system  evolution 


•  manage  future  uncertainty 

•  assure  cost-effective  agility 


The  right  architecture  paves  the  way  for  system  success. 
The  wrong  architecture  usually  spells  some  form  of  disaster. 
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Trends  Leading  to  Architecture  Challenges 


Scale  and  complexity 

Increased  operational 
tempo 

Decentralization  and 
distribution 

Disruptive  technologies 
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Architecture  Challenges  Extend  Across  a 
Spectrum  of  System  Types  and  Scale 


Challenges  include: 

•  determining  how  to  structure  and  adapt  systems  at  all  scales 

•  managing  interactions  among  these  types  of  systems 

•  assuring  software-reliant  capabilities  that  are  sufficiently  reliable,  secure, 
responsive,  and  adaptable  to  change 


Stand-alone  Systems 
software  applications 


Embedded  Systems 

software  embedded 
in  hardware  devices 


Predict  and  control  behavior 


Cyber-Physical 

Systems 

mutually  dependent 
computational 
systems  and  physical 
processes 


Software  Product 
Lines 

families  of  similar 
systems 


Systems  of  Systems  Ultra-Large-Scale 

Systems 

federations  of 

independent  systems  webs  of  software-reliant 

systems,  people, 
economies,  and  cultures 

Assure  and  bound  behavior 


Coupling  to  organizational  structure  and  practices  increases 
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Quality  Attribute  Requirements 


Quality  attributes  include 

•  performance 

•  availability 

•  interoperability 

•  modifiability 

•  evolvability 

•  usability 

•  security 

•  etc. 


Quality  attribute  requirements  stem  from  business  and  mission  goals. 
Key  quality  attributes  need  to  be  characterized  in  a  system-specific  way. 
Otherwise,  they  are  not  operational. 
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Key  Principles  of  Resiliency 


Resilience  is  the  ability  to  provide  and  maintain  an  acceptable  level 
of  service  in  the  face  of  faults  and  challenges  to  normal  operation. 


•  security  “built  in” 

•  failure  scenarios 
understood,  planned  for 

•  redundancy  is  provided 
for  in  key  areas 

•  capability  remains 
available  under  adverse 
conditions 


resilience 


At  SEI,  both  organizational  and  software: 

•  Resilience  Maturity  Model  (RMM) 


•  Security  Quality  Requirements  Engineering  (SQUARE) 


•  Current  blog  series  topic  (http://blog.sei.cmu.edu/) 
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A  key  aim  of  resiliency  (and 
managing  operational  risk) 


Business  Functions: 

•  Developing  and  executing 
continuity  plans,  recovery 
and  restoration  plans 


IT  Function: 

•  Developing,  implementing 
and  managing  processes 
to  deliver  IT  services  and 
manage  IT  infrastructures 


Continuity 
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Resiliency  Maturity  Model 


What  is  CERT-RMM? 

CERT-RMM  is  a  maturity  model  for 
managing  and  improving  operational 
resilience. 

•  Guides  implementation  and  management 
of  operational  resilience  activities 

•  Converges  key  operational  risk 
management  activities:  security,  business 
continuity/disaster  recovery,  and  IT 
operations 

•  Defines  maturity  through  capability  levels 
(like  CMMI) 

•  Improves  confidence  in  how  an 
organization  responds  in  times  of 
operational  stress 
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Key  Ideas  in  Software  Engineering  -1 


Critical  Code — Software  Practice  and  Research 

1 .  There  is  a  rapid  growth  in  the  strategic  significance  of 
software  for  DoD 

-  DoD  needs  to  actively  address  its  software  producibility 
needs 

-  DoD  cannot  rely  on  industry  alone  to  address  software 
challenges  for  defense 

2.  Iterative  engineering  of  innovative  software  can  be 
successfully  managed 

-  Apply  advanced  technologies  and  practice  for  iterative 
incremental  development  of  software  intensive  systems 


a — 


-I  i  ^ 


».  I  \ 


CRITICAL  CODE 

SOFTWARE  PRODUCIBILITY  FOR  DEFENSE 


-  Update  earned  value  models  and  practices  to  support 
management  process 


3.  There  is  insufficient  DoD-aligned  software  experience 


-  DoD  needs  to  be  a  smart  software  customer 
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Key  Ideas  in  Software  Engineering  -2 


4.  Assert  DoD  architectural  leadership 

-  In  highly  complex  systems,  architecture  decisions 
may  need  to  dominate  functional  capability  choices 

5.  Adopt  a  strategic  approach  to  software  assurance 

-  Integrate  preventive  practices  into  development  to 
support  ongoing  creation  of  evidence  in  support  of 
assurance 

6.  It  is  essential  to  reinvigorate  DoD  software  engineering 
research 

-  NITRD  data  reveal  the  extent  of  the  S&T 
disengagement 

-  Apply  appropriate  criteria  in  identifying  goals  for 
research  programs 

-  Focus  research  effort  on  identified  goals  in  seven 
technological  areas 


Software  Engineering  Institute  CarnegieMellon 


Software,  Security,  and  Resiliency 
Paul  Nielsen,  October  6,  201 1 
©  201 11  Carnegie  Mellon  University 


34 


Recommendations  -1 


Summary  of  recommendations 
Improve  critical  areas  of  current  practice 

-  Process  and  measurement 

•  Enable  incremental  iterative  development  at 
arm's  length 

-  Architecture 

•  Enable  architecture  leadership,  interlinking, 
flexibility 

-  Assurance  and  security 

•  Enable  high  assurance  at  scale  with  rich 
supply  chains 
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Recommendations  -2 


Undertake  research  to  support  the  critical  areas 
of  practice 

1.  Architecture  modeling  and  architectural 
analysis 

2.  Validation,  verification,  and  analysis  of  design 
and  code 

3.  Process  support  and  economic  models  for 
assurance 

4.  Requirements 

5.  Language,  modeling,  code  and  tools 

6.  Cyber-physical  systems 

7.  Human-system  interaction 


Software  Engineering  Institute  CarnegieMellon 


Software,  Security,  and  Resiliency 
Paul  Nielsen,  October  6,  201 1 
©  201 11  Carnegie  Mellon  University 


People 


Software  Engineering  Institute 


Software,  Security,  and  Resiliency 

Carnegie  Mellon  Paul  Nielsen,  October  6,  2011 

^  ®  oninramanio  Mallnn  llniuarcih/ 


)  201 11  Carnegie  Mellon  University 


Summary 


Software  has  increasing  impact 
Software  challenges  are  growing 
Software  engineering  provides  useful  tools 
Great  people  are  crucial 
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Format 


Presenter  /  Point  of  Contact 

Title 

Program  or  Initiative 
Telephone:  +1  412-268-5800 
Email:  info@sei.cmu.edu 

Web 

www.sei.cmu.edu 

www.sei.cmu.edu/contact.cfm 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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2011  Highlights 


SEI  Family  of  Models:  Status  &  Plans 


Future  of  the  CMMI  Product  Suite 
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2011  Highlights 


131,664  students  trained  in  Introduction  to  CMMI  as  of 
30  September  2011: 

•  2011  Delivery:  SEI:  198  Partner:  10,133 

•  Total  Delivery:  SEI:  7,596  Partner:  124,068 

Release  of  the  Resilience  Management  Model  (RMM) 

Release  of  the  Smart  Grid  Maturity  Model  (SGMM) 

CMMI  Translations 
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CMMI  Translations 


What’s  New? 

•  New  Translations 

-  CMMI-DEV  VI. 3  Dutch 

-  CMMI-DEV  VI  .3  French 

-  Intro  to  CMMI-DEV  VI  .3  course  French 
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CMMI  Translations 


What’s  Next? 


http://www.sei.cmu.edu/cmmi/solutions/translations/ 


•  CMMI-DEV  V1.3 

-  German 

-  Korean 

-  Portuguese 

-  Simplified  Chinese 

-  Spanish 

-  Traditional  Chinese 

•  CMMI-SVC  VI. 3 

-  Arabic 

•  Intro  to  CMMI-DEV  course 

-  German 


Overview  Why  CMMI  Getting  Started  CMMI  Solutions  CMMI  Compatibility  Research  Our  People 
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CM  Ml  Transition  Status 
Reported  to  the  SEI  as  of  10-31-11 


Training 

Introduction  to  CMMI  DEV  (cumulative/vl  .3) 

131,345/7,485 

Intermediate  Concepts  of  CMMI  (cumulative/vl  .3) 

3,276  /  54 

Understanding  CMMI  High  Maturity  Practices  (cumulative/vl  .3) 

682  /  47 

Acquisition  Supplement  for  Development  (cumulative/vl  .3) 

1,556  /  208 

Services  Supplement  for  CMMI  for  Development 

3231 

Introduction  to  CMMI  for  Services 

1094 

Development  Supplement  for  CMMI  for  Services 

151 

Certifications 

SEI-Certified  Introduction  to  CMMI  for  Development  Instructors  (vl  .2/vl  .3) 

388  /  386 

SEI-Certified  Acquisition  Supplement  for  CMMI  -DEV  Instructors  (vl  .2/vl  .3) 

64/70 

SEI-Certified  Services  Supplement  for  CMMI  -DEV  Instructors 

158 

SEI-Certified  Introduction  to  CMMI  for  Services  Instructors 

88 

SEI-Certified  Development  Supplement  for  CMMI-SVC  Instructors 

42 

SCAMPI-DEV  Lead  Appraisers  (vl  .2/vl  .3) 

421  /321 

SCAMPI  High  Maturity  Lead  Appraisers 

140 

CMMI-ACQ  Lead  Appraisers  (vl  .2/vl  .3) 

76/53 

CMMI-SVC  Lead  Appraisers  (vl  .2/vl  .3) 

171/138 
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Introduction  to  CMMI-DEV  Attendees 
Cumulative  as  of  10-31-11 
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Number  of  SCAMPI  VI  .2/VI  .3,  Class  A  Appraisals 
Conducted  by  Year  by  Representation* 

Reported  as  of  10-31-2011 

*Where  Representation  is  reported 
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2011  Highlights 


SEI  Family  of  Models:  Status  &  Plans 


Future  of  the  CMMI  Product  Suite 


SEI  Family  of  Models 


ENTERPRISE  DATA  MANAGEMENT 

EDM 


COUNCIL 


CMMI 

for  Acquisition 


Guidelines 
for  Improving 
the  Acquisition 
of  Products 
and  Services 


Second  Edition 


Brian  P.  Gallagher  •  Mike  Phillips 
Karen  ).  Richter  *  Sandy  Slirum 
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Guidelines 
for  Process 
Integration 
and  Product 
Improvement 

Third  Edition 

Mary  Beth  Chrissis 

Mike  Konrad 

Sandy  Shrum 

# 


CMMI, 


People  CMM 

Second  Edition 

A  Framework  for 

Human  Capital  Management 


Bill  Curtis 

William  E.  Hefley 

Sally  A.  Miller 
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SCAMPI  Appraisal  Program 


Status 

•  Release  of  SCAMPI  Lead 
Appraiser  Training  (SLAT) 
course 

•  First  delivery  in  September 

-  Feedback  was  positive 


Plans 

•  Merger  of  B&C  handbook 
into  the  Method  Description 
Document  (MDD) 

•  Completion  of  On-line  SLAT 


-  Software  Engineering  Institute 


Standard  CMMI®  Appraisal  Method  for 
Roc  ess  Improvement  (SCAMPI  '  )  A. 
Version  1.3:  Method  Definition  Document 


ECttHFI  Liptr^DE  Tfcani 

Hare*  ED1 1 

HANDBOOK. 
CMU<£E-2ai  1-HE-H1 


E-iglrwrlm;  Manausn’wt 
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SCAMPI  Appraisals  -  Organization  Type 


4000 


3500 

3000 

2500 

2000 

1500 

1000 

500 

0 


Commercial/ 

Internal-Use 


Contractor  (for 
Military/Government) 


Military/Government 

Agency 


Number  of  Organizations 


3397 


667 


156 
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Size  of  Appraised  Organizations 


501  to  1000 
4.0% 


1001  to  2000 
1.9% 


2000+ 

1.6% 


301  to  500 
5.4% 


201  to  300 
5.8% 


201  to  2000+ 
18.6% 


25  or  fewer 
20.3% 


101  to  200 
15.1% 


1  to  100 
66.3% 


76  to  100 
7.6% 


26  to  50 
25.9% 


51  to  75 
12.6% 


Software  Engineering  Institute  CarnegieMellon 
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Appraisal  Maturity  Levels  Audited  or  Reviewed 


0  1  2  3  4  5  B/C  or  Review 


■  2007 

■  2008 

■  2009 

■  2010 
■  2011 
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CMMI-ACQ 


Status 

•  Alex  Stall  has  accepted  the 
product  manager  role  for  CMMI- 
ACQ 

Plans 

•  Define  the  market(s) 

•  Define  what  acquisition  is  to  the 
non-DoD  market 

•  Increase  interest  and  adoption 


CMMI 

for  Acquisition 

Guidelines 
for  Improving 
the  Acquisition 
of  Products 
and  Services 

Second  Edition 

Brian  P.  Gallagher  *  Mike  Phillips 
Karen  J.  Richter  •  Sandy  Shrum 


Software  Engineering  Institute  CarnegieMellon 
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CMMI-SVC 


Status 

•  More  than  130  appraisals  reported 

•  25%  increase  quarter  over  quarter  in 
appraisals 

•  Documented  ROI  examples  from  partners 

•  Five  graduate  dissertations  /theses  have 
been  written  on  CMMI-SVC 

•  Two  books  by  partners  featuring  CMMI- 
SVC  have  been  published,  and  a  third  book 
by  a  partner  is  under  way 

•  Over  100  examples  submitted  about  SVC 
implementation 

Plans 

•  Advanced  SVC  training 

•  Produce  guidance  on  high  maturity  in 
service  settings 

•  Evaluating  new  pricing  and  certification  path 

•  Focus  sessions  for  Partner  input 


-  Software  Engineering  Institute 


Ca 


CMMI-SVC,  Version  1.3 


CMMI 
for  Services 


Guidelines 

for 

Superior 

Service 


Second  Edition 


Eileen  C.  Forrester 
Bomdon  L.  Butcim 
Sandv  Shrum 
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CMMI-DEV 


Status 

•  Ten  CMMI  translations  on-going 

•  CMMI  courses  were  updated 

•  Intro  to  CMMI  vl  .2  will  be  retired  on 
November  30,  201 1 

•  95%  of  the  CMMI-DEV  instructors  have 
upgraded  to  VI. 3 

Plans 

•  Incorporate  Lean  and  Agile  principles 

•  Devise  lightweight,  cost  effective  solutions 

•  Selected  training  to  eLearning 

•  Re-focus  use  of  CMMI  on  achieving 
superior  attention,  motivation,  and 
cooperation  to  improve  performance 


=  Software  Engineering  Institute 


SEI  5  ERIC  5  IN  SOFTWARE  ENGINEER  INQ 


CMMl  -DEV,  Version  1.3 


CMMI 

for  Development 

Guidelines 
for  Process 
Integration 
m  and  Product 
I  Improvement 


fl 

■ 

■ 

Third  Edition 


Mary  Beth  Chrissis 

Mike  Konrad 

- - 

Sandy  Shrum 
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People  CMM 


Status 

•  Ongoing  revisions  to  model 
aimed  at  version  2 

Plans 

•  Continue  working  on  change 
packages  for  the  next  version 
of  the  People  CMM  which  will 
bring  the  People  CMM  in  line 
with  VI  .3  ofCMMI 

•  SEI  People  CMM  SCAMPI 


!  People  CMM 

Second  Edition 


A  Framework  for 
Human  Capital  Management 


Bill  Curtis 
William  E.  Heflcy 
Sally  A.  Miller 
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Smart  Grid  Maturity  Model 


Status 

•  Creation  of  a  management  tool  for  the 
utility  industry  Smart  Grid  transformation 

•  Utilities  need  to  use  the  SGMM  to: 

•  Develop  effective  roadmaps 

•  Track  progress 

•  Understand  their  posture  in  comparison  to 
peers 

•  SEI  Training  and  licensing  available 


=^=-  Software  Engineering  Institute  CarnegieMellon 


)2011  Carnegie  Mellon  University 


19 


CERT 


Management  Model  (RMM) 


Status 


•  Model  vl.1  and  appraisal  method  published 

•  Introductory  courses 

•  Model  training 

•  “How-to”  CERT  courses 

•  Executive  workshops 

•  Advanced  courses 

•  User  Groups 

•  Practitioner  training 

•  Appraisal  leader  training 

•  Instructor  training 


Plans 

•  Certifications 

-  Lead  Appraiser 

-  Navigator 

-  Coach 

-  Instructor 


Software  Engineering  Institute  CarnegieMellon 
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Data  Management  Maturity  Model  (DMM) 


Status 


•  Created  in  collaboration  with  data 
practitioners  of  the  financial  information 
industry 

•  Focus  on  how  to  turn  ‘art  and  practice’ 
into  ‘science  and  discipline’. 

Plans 

•  The  final  output  will  be  a  framework 
document  and  accompanying 
assessment  methodology  (SCAMPI)  for 

•  Evaluating  the  efficiency  of  data 
management  practices 

•  Measuring  the  maturity  of  operational 
integration 

•  Establishing  best  practices  that  can  be 
adopted  in  varying  domains  worldwide 
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Today’s  Discussions... 

2011  Highlights 


SEI  Family  of  Models:  Status  &  Plans 


Future  of  the  CMMI  Product  Suite 


=  Software  Engineering  Institute 


Can 


“  The  best  way  to  predict  the  future  is 

to  invent  it\” 


Alan  Kay,  1971,  American  computer  scientist,  researcher,  and  visionary 
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Software  Has  Come  of  Age 


“...we  are  in  the  middle  of  a  dramatic  and  broad  technological 
and  economic  shift  in  which  software  companies  are  poised 
to  take  over  large  swathes  of  the  economy.  ” 


--Marc  Andreessen 

“Why  Software  Is  Eating  the  World” 

The  Wall  Street  Journal,  August  2011 


Transforming  into  “software”  companies: 


Healthcare 
Music  companies 
Telecom 
Entertainment 
Companies 


Booksellers 
Movie  production 
Financial  services 
Education 
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CMMI  &  TSP  are  Ranked  Top 
Software  Engineering  Best  Practices! 


Methods 

1)  Agile 

2)  TSP/PSP 

3)  Waterfall 

4)  CMM1 1,2 


Methods 

1)  TSP/PSP 

2)  Agile 

3)  CMMI  3 

4)  RUP 


Methods 

1)  TSP/PSP 

2)  CMMI  3,  4,  5 

3)  RUP 

4)  Hybrid 


Development  practices  by  size  of  application^ 


[1]  Software 

Engineering  Best 
Practices,  by  Capers 
Jones,  2010. 

Based  on  a 
comprehensive 
study  of  630 
organizations  and 
13,000  projects. 
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SEI  Response:  Broader  Outreach 


Responsibility  for  improving  software  development  process 


Everyone 
Process  group 
Project  managers 

Development  team 

Practitione  rs  (e.g  ,r  d  evelope  rs, 

testers) 

Executive  management 
Mo  one 

Community 

I  don't  know 

Base:  81  IT  professionals 


32% 


19% 

15% 

14% 


9% 


6% 

4% 


1% 

1% 


Source:  2009  Global  Software  Development  Process 
Online  Survey 


Shifting  SEI  process  improvement 
brand  messaging  to  multiple  targets  in 
an  organization 
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New  Directions:  Multi-Model  Implementations 


“Forrester  Research’s  survey  of  Dr.  Dobb’s  readers  underscores  what  the  SEI  and 
Partners  already  know:  software  development  process  is  a  multi-model  world.  ” 

-SEI  Market  Insights  Research  Report 


The  SEI  is  developing  a  common  language  for  development  and  a 
way  to  attain  and  measure  improvement  in  multi-model  environments. 


CECOM  Application 

•  CECOM  Software  Engineering  Center  is  applying  CMMI- 
DEV,  CMMI-ACQ,  and  CMMI-SVC  to  meet  its  complex 
needs. 

•  “Hybrid”  model  approach  offers  each  Mission  Area 
Directorate  to  concentrate  on  their  specific  process 
improvement  needs. 
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CMMI:  What’s  Next?  - 1 


New  Positioning: 


•  Drive  down  the  costs  of  using  CMMI  by  providing  innovative 
technology-supported  solutions 

•  On-line  tools  for  CMMI  Assessments/Appraisals 

•  Contextual  help  for  key  processes  and  key  practices 

•  eLearning  (bite-sized  CMMI  training  segments  delivered  just 
in  time) 

•  manage  adoption  of  disruptive  technologies  (e.g.,  mobile 
apps,  cloud,  social  networks, ...) 


-  Software  Engineering  Institute  CarnegieMellon  ©2011  Carnegie  Mellon  University 


CMMI:  What’s  Next? -2 


•  Marketing/Imaging/Messaging: 

-  Initiate  faster,  better,  cheaper  thinking  without 
sacrificing  quality  and  brand  image 

-  Market  our  products  and  services  as  an  expert’s 
toolkit 


•  Consider  approaches  to  Small  and  Medium 
Enterprise  market  (low  cost,  high  volume,  high 
automation) 

•  Investigate  marrying  CMMI  with  more  performance- 
oriented  approaches 

•  Address  Multi  model  integration  for  SEPM  and  NON- 
SEPM  products  and  performance  measures 


Program  Direction 


Better  integration  of  SEI  process  and 
measurement  products  and  efforts. 


Development  of  next-generation  integrated 
software  models,  methods,  and  practices. 


Increased  focus  on  the  transition  and  application 
of  SEI  solutions  to  the  problems  of  industry  and 
the  world. 


Increased  research  focus  on  software  engineering 
methods,  models,  and  measurement  practices 


One  Vision  of  the  Future  CMMI  Product  Suite 

Process  Integration 

•  Models,  Methods,  Standards 

-CMMI,  ISO,  COBIT,  Six  Sigma,  IT  Mark,  TSP,  AIM,  CENELEC, 
RMM,  SGMM,  Lean,  and  Agile,  etc. 

•  Dynamically  build  models/hybrids  of  other  models,  methods,  and 
standards 

-  Use  only  the  practices/parts  the  organization  needs 
-Build  the  appraisal  rules 

•  Performance  data  submissions  as  a  part  of  appraisals 

Project  and  Program  Management  using  High  Maturity  Techniques 
Coach/Mentor  Credentials  for  Process  Improvement 
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Moving  to  the  Future 


We  need  to  move  the  technology  forward. 
We  need  to  capitalize  on  our  strengths: 

•  Model  technology 

•  Appraisal  technology 

•  Measurement  technology 

•  Process  &  engineering  technologies 


Models, 

Methods, 

Standards 


-TSP 

-Architecture 
•  T rusted  broker 
-SAS 
-PARS 


p 

M 

erformanc 

easureme 

;e 

nt 

Appraisal 

Rules 
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Concept  -  Models 


Database 

•  Hybrid  model 

-  Tree  structure 

-  Model  references 

-  Interpretational/ 
implementation 
rules 

•  Performance 

-  Work  sampling 

-  Standard  set  of 
measures 

•  Appraisal  rules 

-  Generic  for  whole 
model 

-  Specific  for 
individual  practices 


Database 


Hybrid  Model 


Appraisal 

Rules 


Generic 

Appraisal 


Performance 


Specific 

Practice  or  Specific  Implementation 

Pointer  Measures  or  Appraisal 

Guidance 


Questions, 

Look/Listen 

Fors 


Sampling 


Standard 

Measures 


Generic  appraisal 


Implementation  quick  start 
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CMMI 


Hybrid  Models,  Appraisal  Rules  & 
Generic  Appraisals 

Models,  standards, 
methods,  etc. 

•  Existing 

-CMMI-ACQ,  DEV,  SVC 
-  RMM 


RMM 


Dynamic 


r  h 
l\ 

tybrii 

yiode 

d  1 
>1  . 

-  SGMM 

-  TSP 

-  ISO  NNN 

-  Six  Sigma 
Dynamic 

-  From  literature 

-  Constructs  based  on 

existing  systems  or  iS0 

implementations 

•  Agile 


Lean/Agile 
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For  More  Information 


Rusty  Young 

CMMI  Program  Manager 

412-268-2584 

rrv@sei.cmu.edu 


-  Software  Engineering  Institute 


Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  412-268-5800 
SEI  Phone:412-268-5800 
SEI  Fax:  412-268-6257 


Carnegie  Mellon 
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□Acquisition 

REFORM 


■■IV 

HP 

s 

CMMI®for  Acquisition  and  CMMI®for  Development: 
Potential  Favorable  Contributors  to  the  Rapid 
Acquisition  of  IT  Systems  in  Support  of 

Public  Law  111 


1 1 th  Annual  CMMI  ®  Technology  Conference  and  User  Group  Kenneth  E.  Nidiffer 

National  Defense  Industrial  Association  Software  Engineering  Institute 

November  14-17,  2011  Carnegie  Mellon  University 

Pittsburgh,  PA  15213 
703-908-1117 


Software  Engineering  Institute 


Carnegie  Mellon 
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Overview 


•  Background 

•  Overview  of  Proposed 
Changes 

•  Challenges 

•  Summary 


This  presentation  focuses  on  the 
past,  present,  and  future  of  IT 
acquisitions 


Source:  SEI 


A  fl. 

'm-'±  m 

iiMjb 
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What  is  the  Information  Technology  (IT)  Environment? 


Includes  all 

•  System  of  Systems 

•  Architecture 

•  Services 

•  Networked 
Hardware/ 
Platforms 

•  People  who 
digitally  connect  to 
cyberspace 


Source:  SEI 


Often  difficult  to  distinguish  IT  systems  from  other 
types  of  systems  because  they  are  networked  systems! 
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Examples  of  Emerging  Ultra-Large  IT  Systems 
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Optimizing  Information  Technology* 


•  Adaptable  IT  Infrastructure  Center  on 

-  Cloud  Computing 

-  Common  Services 

-  Network/Desktop  Consolidation 

•  Streamlined  IT  Acquisition  Processes 

•  Robust  Cyber  Security  Implementation 

•  Information  Sharing  Approaches 

•  Information  Assurance 

•  Renewed  Focus  on  Using  Facilities  on  Reducing  Overall 

Environment  Consumption  (e.g.,  power,  space,  cooling) 

Sources:  Department  of  Defense  (DoD)  Chief  Information  Officer  (CIO) 

*  Partial  List  Campaign  Plan  Baseline  (Oct,  2011)  and  SEI  Discussions  with  CIOs  (2010) 
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Acquisition:  IT  Systems  are  Different  from  a  Weapon  System  — 
and  Critical  to  Enable  a  more  Resilient  Cyber  Environment 


Weapon  Systems 


•Weapon  platform  centric 

•Military  unique  requirements 

•  Development  of  military- 
unique,  breakthrough 
technologies 

•  Development  cycle  of  decade 
or  more 

•  Production  decisions  for 
unique  hardware 

•Service  lives  extending  into 
decades 


f 


IT  Systems 

•  Enterprise  network 
centric 

•Adapt  commercial 
capabilities  for  military  needs 

•  Leverage  commercial 
technologies 

•Technology  cycle  12-18 
months 

•  Procure  commodity  hardware 

•  Periodic  technology  refresh  to 
avoid  obsolescence 


Demand  Different  Acquisition  Life  Cycles  and  Processes 


Sources:  IT  Acquisition  Reform  Task  Force/M ITRE  Corporation 
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DoD  IT  Acquisition  Cycle  Time  -  32  MAIS 


Milestone  B 
Planning  Phase 


Build  Phase 


Initial 

Operational 

Capability 


Analysis  of  Economic 

Alternatives  Analysis 


Development 

—  40 - 


Cycle  Time  Driven  by  Processes  Developed  to  Counter 
a  Cold  War  Adversary  In  Industrial  Age  Society 


Source:  Defense  Science  Board  Report,  March  2009 
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The  Opportunities 

Adaptability 


Speed 


Agility 


MAIS  Program  Avg  =  91  Months 


>nn  established  a  new  goal:  18  months 


Source:  A  New  Approach  for  Delivering  IT  Capabilities  in  DoD,  Report  to  Congress,  November  2010 
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IT  Acquisition  Reform  Imperative 


Congress 

•  Develop  and  Implement  a  new  process  for  Acquiring  IT 

(FY10  NDAA*  Section  804) 

•  HASC**  Panel  on  Defense  Acquisition  Reform  Finding  and 
Recommendations  (23  March  2010) 


Widely,  documented 

Problems  with  DoD  IT 

Acquisitions 

•Defense  Science  Board 
-Jan  09  -  Integrating  COTS 
-Mar  ‘09  -  IT  Acquisition 
-Apr  ‘09  -  Fix  the  Acq  process 
-Jul  ‘09  -  Rapid  Acquisition 
•Industry  Associations 
-AFEI,  TechAmerica, 

•National  Academies  -  Achieving  Effective 
Acq  of  IT  in  DoD  2010 

•Business  Leads  -  Aug  ‘08  Joint  DISA  IT 
Review 


Federal  CIO 

25-Pt  Implementation  Plan 
to  Reform  Federal  IT 
Management 

Vivek  Kundra,  U.S.  CIO, 
December  9,  2010 


DoD  Senior  Leadership  Vision 

“First  step  [for  DoD  to  succeed  in  delivery  of  IT]  is  to  acknowledge  that  simply  tailoring 
the  existing  processes  in  not  sufficient ”  (National  Research  Council,  DEC  2009) 


NDAA:  National  Defense  Authorization  Act ;  HASC:  House  Armed  Services  Committee; 
AFEI:  Association  for  Enterprise  Information;  DISA:  Defense  Information  Systems  Agency 
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Defense  Science  Board  Report  &  Public  Law  111 
(Section  804)  2010  National  Defense  Authorization 
Act 

Selected  Statements  (paraphrased): 

>NEW  ACQUISITION  PROCESS  REQUIRED  -The 
Secretary  of  Defense  shall  develop  and  implement  a  new 
acquisition  process  for  information  technology  systems 

>  To  the  extent  determined  by  the  Secretary,  be  based  on 
the  recommendations  in  Chapter  6  of  the  March  2009  report 
of  the  DSB  Task  Force  on  DoD  and  Procedures  for  the 
Acquisition  of  Information  Technology 
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A  New  Acquisition  Process  for  Information 
Technology 


Milestone  Build 
Decision 

0  Materiel  a  A 

Development  O  /\ 

Decision  V  “  RELEASE  1  V 


Business  Case  Analysis 
and  Development 

Architectural  Development 
and  Risk  Reduction 

Development  &  Demonstration 

Operations 

and 

Prototypes 

Iterationl 

Iteration  2 

Iteration  “N' 

Support 

* - Coordinated  DOD  stakeholder  involvement  ■>  <  Integrated  DT  /  OT  > 


< -  Up  to  2  years  - 

0 

- ^ 

0 

RELEASE  2 

Prototypes 

Development  & 
Demonstration 

Operations 

iteration  1  IterittorallttfittonJ 

and  Support 

ICD  Initial  Capability  Document 

0 

0 

CDD  Capabilities  Development  Document 

RELEASE  "N" 

Prototypes 

Development  4 
Demonstration 

Operations 

Decision  Point 

IttratioH  llUiratlon  2  llisralldn  3 

and  Support 

L 

Continuous  Technology/Requirements  Development  &  Maturation 


Source:  Defense  Science  Board  Report,  March  2009 
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Guiding  Principles 


1.  Deliver  Early  and  Often 

2.  Incremental  and  Iterative  Development  and  Testing 

3.  Rationalized  Requirements 

4.  Flexible/Tailored  Processes 

5.  Knowledgeable  and  Experienced  IT  Workforce 

Sources:  Defense  Science  Board  Report,  March  2009  and  A  New  Approach  for  Delivering  IT 
Capabilities  in  DoD,  Report  to  Congress,  November  2010 
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DOD  Software  Acquisition  Process  Improvement  Programs,  DoD 
Major  Events,  and  Leadership  Rotation 


Cohen  1 

Rumsfeld 

1  Gates 

01-24-1997  to  01-20-2001  I 

01-20-2001  to  12-17-2006 

1  12-17-2006  to  Present 

Pre-Public  Law  Actions-SSWG 

I 


2000 


I 


2001 


2002 


Army  Strategic 
Software  Improvement 
Program 
Aug  18,  2002 


OSD  Software 
Intensive  Systems 
Steering  Group 
(SISSG) 

June  6,  2000 


GAO  Audit 
Reports 


2003 


OSD  Policy  for 

Systems  Engineering 
Forum  Established 

Feb  2,  2004 

DoDD  5010.42 
CPI/LSS 

May  15,  2008 

A 

L  L  I 


A  A 


V  V 


2004  A 

I* 

A 


2005 


2006 


2007 


2008 


2009 


I  I  I  I  *  I 


OSD  Guidance 
on  Section  804 
Mar  21,  2003 


OSD  Software  Intensive 
Systems  Steering 
Group  (SISSG) 

June  11,  2003 


Air  Force  Software- 
Intensive  Systems 
Strategic  Improvement 
Program 
Jan  13,2004 


Navy  Software 
Process 
Improvement 
Program 
May  15,2006 


DoD/NSA  Establish 
Systems  Engineering 
Research  Center 
(SERC)  Nov  24,  2008 


A 

■ 

OSD  GAO  Response 

OSD  GAO  Response 

■ 

Dec  21,2004 

Dec  1,  2006  Follow-up 

■ 

■ 

Follow-up  will  be 

Section  804  met  under 

■ 

■ 

though  SE 

SE  Revitalization 

■ 

■ 

■ 

Revitalization 

No  further  reporting. 

W 

A 

A 

V 


V 


DOD  Software 
Acquisition  Training  and 
Education  Working  Group 
Feb  2,  2008 


Declared  end  of  OSD 
Reporting  on  PL  107-314 


OSD  Tri-Service 
Assessments 
Initiative 
1999-2003/4 


Public  Law 


GAO-01  -11 6 

Section  804 

GAO-04-393 

_ ■ _ 

GAO-04-393 

_ ■ _ 

GAO-04-393 

GAO-09-888 

Mar  2001 

Dec  2,  2002 

Report 

Follow-up 

Follow-up 

Sept  2009 

Report 

July  2004 

Report 

Report 

Report 

Source:  OSD/AT&L 

July  2005 

July  2006 

Software  Improvement  Focus 


Systems  Engineering  Improvement  Focus 
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An  Effective  Process  for  Major  Defense  Systems  - 
but  not  very  agile  for  IT  Systems 


I  nt  eg  rated  Defense  Acquis  Eta  on,  Technology,  and  Logistics  Life  Cycle  Management  System 
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Better  Alignment:  Three  Major 
DoD  Decision  Support  Systems 


Planning, 
Programming, 
Budgeting  & 
Execution 
(PPBE) 


Effective  Interaction 
Essential  for  Success 


Joint  Capabilities 
Integration  & 
Development 
System 
(JCIDS) 


Defense 
Acquisition 
System 


Big  A 


Source:  Defense  Acquisition  University 
Aug, 2010 
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Challenges:  Information  Technology  (IT)  Environment 


Source:  www.sei.cmu.edu/uls 
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Rate  of  Adoption 

The  Cyber  Domain  is  Hotly  Contested 


High 


C 

O 

(0 

o 

.52 

Q. 

O 

w 


Low 


Sophistication 
Required  of  Actors 
Declining 

~7 


Sophistication 
Of  Available  Tools 
Growing 


sophisticated  C2 
cross  site  scripting 


“stealth”  /  advanced 
scanning 
techniques 

denial  of 
service 


...next? 


Staging 


distributed 
attack  tools 
www  attacks 
automated  probes/scans 


back  doors 
disabling  audits 


burglaries 

'exploiting  known  vulnerabili 

!  password  cracking 

self-replicating  code 
password  guessing 


network  mgmt.  diagnostics 

hijacking 

sessions 


Increased  GIG  Complexity 
and  Dependence  equates 
to  lower  entry  barriers  and 
potential  for  increased 
number  of  malicious  actors} 


Source:  DoD 


Defensive  measures  are  outpaced  by  the  well  resourced  sophisticated  threat . . . 
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Agile  Development  Process: 

A  Building  Block  for  Constructing  Capabilities 


Slory  boarding 


Product  Backlog 
□□□□□□□ 
□□□□□□□ 
□□□□□□□ 
□□□□□□□ 
□□□□□□□ 
□□□□nan 
□□nnnnn 

Product 

Backlog 

(Requirements 

Generation) 


Continuous  Integration  and  Test 

♦  Developmental 

•  Operational 

*  Interoperability 

♦  Security 


Iteration 

Backlog 

,  □□□□ 
□  □□□ 
□  □□□ 


Voice  of  the  User 


User  Stories  1— ILJ— 11— I  Test  Cases 

Iteration 

Backlog 

(Highest  Priority 
User  Requirements) 


Iteration 


(Capability 

Development) 


Iteration 

Review 

(Deployment 

Decision) 


Working  Product 

(Militarily  Useful  Capability) 


7] 


Agile  methods  emphasize  early  and  continuous  delivery  of 
valuable  products  and  services 

•  User  immersion  in  development  for  clarification  and  feedback 

•  Iterative  development 

•  Testing  occurs  throughout  (instead  of  at  the  end) 

•  Working  product  is  the  primary  measure  of  progress 


Sources:  IT  Acquisition  Reform  Task  Force/M  ITRE  Corporation 
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Agile  is  Not  One-Size-Fits-AII 


Incremental  plans  the  content  of  multiple  releases 


*A  Release  is  comprised  of  two  or  more  iterations. 


Note:  The  diagram  which  was  created  by  the  MITRE  Corporation  team  on  the  IT  Acquisition  Task  Force 
Initiative  is  only  for  illustrative  purposes  and  not  necessarily  representative  of  any  specific  agile  approach 
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Assessment  Framework:  CMMI®-ACQ 


Operational  Need 


Focus  on  Acquisition  Best  Practices 

(Acquirer) 


CMMI®  -ACQ  Categories  and  Process  Areas 


Category 

Process  Area 

Acquisition 

Agreement  Management  (AM) 

Acquisition  Requirements  Development  (ARD) 

Acquisition  Technical  Management  (ATM) 

Acquisition  Validation  (AVAL) 

Acquisition  Verification  (AVER) 

Solicitation  and  Supplier  Agreement  Development  (SSAD) 

Process 

Management 

Organizational  Innovation  and  Deployment  (OID) 
Organizational  Process  Definition  (OPD) 

Organizational  Process  Focus  (OPF) 

Organizational  Process  Performance  (OPP) 

Organizational  Training  (OT) 

Project 

Management 

Integrated  Project  Management  (IPM) 

Project  Monitoring  and  Control  (PMC) 

Project  Planning  (PP) 

Quantitative  Project  Management  (QPM) 

Requirements  Management  (REQM) 

Risk  Management  (RSKM) 

Support 

Causal  Analysis  and  Resolution  (CAR) 

Configuration  Management  (CM) 

Decision  Analysis  and  Resolution  (DAR) 

Measurement  and  Analysis  (MA) 

Process  and  Product  Quality  Assurance  (PPQA) 

CMMI®  -ACQ  model  was 
developed  to  codify  best 
practices  to  help  organizations 
improve  acquisition  processes 

CMMI®  reference  models  have 
gained  significant  traction 
across  commercial  and  defense 
community  and  are  widely  used 
throughout  world  [CMMI 
Product  Team  07] 


Source:  SEI 
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CMMI®-ACQ  (VI. 3)  Project  Planning  -  Comparison* 


Project  Planning 

PP 

CMMI  Pratice 

Scrum  Practice 

SG  1 

Estimates  of  project  planning 
parameters  are  established  and 
mai  ntai  ned 

SP  1.1 

Establish  and  maintain  the  acquisition 
strategy 

Concept  of  Operations  or  Scrum  product  vision 

SP  1.2 

Establish  a  top-level  WBS  to  estimate 
the  scope  of  the  roject 

The  standard  tasks  used  in  a  Scrum  process 
combined  with  specific  project  tasks  (Scrum 
Backlog) 

SP  1.3 

Establish  and  maintain  estimates  of  the 
attributes  of  the  work  products  and 
tasks 

Story  points,  used  to  estimate  the  difficutly  (or 
relative  size)  of  a  Story  (requirement) 

SP  1.4 

Define  the  project  lifecycle  phases  on 
which  to  scope  the  planning  effort 

The  Scrum  Process 

SP  1.5 

Estimate  the  project  effort  and  cost  for 
the  work  products  and  tasks  based  on 
estimation  rationale 

Scrum  time  estimate  (similarto  billable  hours  or 
Full  Time  Equivalents) 

SG  2 

A  project  plan  is  established  and 
maintained  as  the  basis  for  managing 
the  project 

SP  2.1 

Establish  and  maintain  the  project's 
budget  and  schedule 

1.  Scrum  estimates;  2.  Estimates  of  what  work  will 
be  in  each  release;  3.  Sprint  Backlog;  4.  Project 
Taskboard 

Reference:  N.  Potter  and  M.  Sakry,  Implementing  Scrum  (Agile)  and  CMM I  ®  Together,  March  2009 
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IT  Compared  with  Other  Sciences 


PHYSICAL  SCIENCE 

BIOSCIENCE 

IT-  COMPUTER/SOFTWARE/CYBER 

SCIENCE 

Origins/History 

Begun  in  antiquity 

Begun  in  antiquity 

Mid-20th  Century 

Enduring  Laws 

Laws  are  foundational  to 
furthering  exploration  in 

the  science 

Laws  are  foundational  to 
furthering  exploration  in  the 

science 

Only  mathematical  laws  have  proven 
foundational  to  computation 

Framework  of 

Four  main  areas: 

Science  of  dealing  with 

•  Several  areas  of  study: 

Scientific  Study 

astronomy,  physics, 
chemistry,  and  earth 

sciences 

health  maintenance  and 

disease 

prevention/treatment 

computer  science,  software/ 
systems  engineering,  IT,  HCI, 
social  dynamics,  Al 
•  All  nodes  attached  to/relying  on 
netted  system 

R&D  and  Launch 

Cycle 

10-20  years 

10-20  years 

Significantly  compressed;  solution 
time  to  market  needs  to  happen 
very  quickly 

Source:  SEI 

HCI:  Human  Computer  Interaction;  Al:  Artificial  intelligence 
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Opportunity  -  21st  Century  Acquisition  Workforce 


■  Acquisition  Guidance,  and  Training  Programs  Need  to  Be  Updated  to 
Support  the  21st  Century  Acquisition  Workforce 


-Traditional  Gonoration 


~Baby  Boomers  - Genoration  X  Generation  Y  - Millenium 


Si  font  General  iork 
1  928-1945 


Baby  Boomers 
1946*1964 


Generation  X 
1965*1980 


Generation  YYNti  Honda  Is 

1981*2000 


Hard  worker 
Respects  authority 
Work  is  obligtatiort 
Formal  communicator 
Work/family  separation 


Workaholic 
Questions  authority 
Works  efficiently 
Competitive 
(Little  workyiife  balance 


Technica  lly  advanced 
Prefers  i  nf o rmalrt y 
Meeds  structure  and  direeUon 
Direct/!  mmed  ia<ts-  -com  mu  ni-cartor 
Seeks  work/life  balance 


Technically  sewy 
Embraces  diversity 
Requires  supervision 
Indirect/  virtu  el  communicator 
Demands  work/I ife  balance 


Sources:  DoD  (AT&L)  and  SEI 
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Why  are  Software  Intensive  IT  Projects  Difficult? 


According  to  Fred  Brooks  software  projects  are  difficult  because 
of  accidental  and  essential  difficulties 

•  Accidental  difficulties  are  caused  by  the  current  state  of  our 
understanding 

-of  methods,  tools,  and  techniques 
-of  the  underlying  technology  base 

•  Essential  difficulties  are  caused  by  the  inherent  nature  of 
software 

-invisibility  -  lack  of  physical  properties 

-conformity 

-changeability 

-complexity 

Source:  The  Mythical  Man-Month  by  Fred  Brooks,  Addison  Wesley,  1995 
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Realities  of  Software  Quality 


The  flowchart  might  correspond  to  a  100 
LOC  module  with  a  single  loop  that  may  be 
executed  no  more  than  20  times. 


There  are  approximately  1014  possible  paths 
that  may  be  executed! 


For  any  but  the  smallest  programs,  complete 
path  coverage  for  defect  detection  is 
impractical. 


Lehman  Laws: 

1 .  The  Law  of  Continuing  Change  -  programs  must  change  to  be  useful 

2.  The  Law  of  Increasing  Complexity  -  programs  that  change  become  more  complex 


Source:  Adapted  from  Pressman,  R.S.,  Software  Engineering:  A  Practitioner’s  Approach ,  Third  Edition,  McGraw  Hill,  1992 
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Software  Evolution  and  Maintenance  Cost  Is  Increasing 


Year 

Proportion  of 
software 

maintenance  costs 

Definition 

Reference 

2000 

>90% 

Software  cost  devoted  to  system  maintenance  & 
evolution  /  total  software  costs 

Erlikh  (2000) 

1993 

75% 

Software  maintenance  /  information  system  budget 
(in  Fortune  1000  companies) 

Eastwood  (1993) 

1990 

>90% 

Software  cost  devoted  to  system  maintenance  & 
evolution  /  total  software  costs 

Moad  (1990) 

1990 

60-70% 

Software  maintenance  /  total  management  information 
systems  (MIS)  operating  budgets 

Huff  (1990) 

1988 

60-70% 

Software  maintenance  /  total  management  information 
systems  (MIS)  operating  budgets 

Port  (1988) 

1984 

65-75% 

Effort  spent  on  software  maintenance  /  total  available 
software  engineering  effort. 

McKee  (1984) 

1981 

>50% 

Staff  time  spent  on  maintenance  /  total  time  (in  487 
organizations) 

Lientz  &  Swanson  (1981) 

1979 

67% 

Maintenance  costs  /  total  software  costs 

Zelkowitz  et  al.  (1979) 

Source:  Jussi  Koskinem,  Department  of  Computer  Science  and  Information  Systems,  University  of  Jyvaskyla 
P.O.  Box  35,  40014  Jyvaskyla,  Finland 
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Federal  IT  Market  Growth 


“In  the  next  five  years,  IT  contractors  will  see  the 
federal  market  for  their  services  increase  by  a 
compound  annual  growth  rate  of  5.4  percent  to  a 
total  of  $11 1.9  billion  by  2015.” 


--  Ben  Bain 

Federal  Computer  Week 
April  8,  2010 
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Proposed  Changes:  Think  and  Perform  Differently 


CMMI®  for  Acquisition  and  CMMI®  for  Development  Can  Have 
a  Potential  Impact  on  the  Rapid  Acquisition  of  IT  Systems 


Source:  Wikipedia-  IBM  Watson:  The  Face  of  Watson  on  You  Tube 
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Summary 


•  The  IT  Acquisition  Task  Force  Initiative  concepts  may  have  to 
be  quite  radical  to  meet  the  IT  Acquisition  Reform  Guiding 
Principles  (Section  804)* 

-  Deliver  Early  and  Often  -  Be  responsive  to  the  users  needs 

-  Incremental  and  Iterative  Development  and  Testing 

-  Rationalized  Requirements  -  Balance  user  needs  with 
constraints 

-  Flexible/Tailored  Processes  -  Customize  to  IT  category 

-  Knowledgeable  and  Experience  IT  Workforce  -  Understands 
IT  uniqueness 

•  CMMI®  for  Acquisition  and  CMMI®  for  Development  Can  Have 
a  Potential  Impact  on  the  Rapid  Acquisition  of  IT  Systems 

•  General  acknowledgement  that  we  can  and  must  do  better! 


Source:  A  New  Approach  for  Delivering  IT  Capabilities  in  the  DoD,  Report  to  Congress,  November  2010 
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Wrap  Up 
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Contact  Information 


Dr.  Kenneth  E.  Nidiffer,  Director  of  Strategic  Plans  for  Government 
Programs 

Software  Engineering  Institute,  Carnegie  Mellon  University 
Office:  +  1  703-908-1117 
Fax:  +  1  703-908-9317 

Email:  nidiffer@sei.cmu.edu 
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SCAMPI  Planning  with  Version 

1.3 


Lynn  Penn 
Dorna  Witkowski 


LOCKHEED 


NDIA-  November,  2011 
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Agenda 


*  Overview  of  Organization 

*  Getting  Started 

*  Sampling  Factors 

*  Sub-Groups 

*  Support  Functions 

*  End  Result  -  So  Far 

*  Lessons  Learned 
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Overview  of  Organization 


•  Information  Systems  and  Global  Solutions  (IS&GS)  - 
Business  Area  within  Lockheed  Martin  Corporation 

-  IS&GS-Defense  Product  Line 

•  Employs  over  12,000  people  at  more  than  200  sites 
worldwide 

•  Principally  engaged  in  the  design,  development, 
operation  and  sustainment  of  systems  and  solutions 
that  help  defense  customers  achieve  their  missions. 

•  Performs  on  more  than  400  programs  for  customers 
that  include  the  U.S.  military  services,  NASA  and  the 
National  Oceanic  and  Atmospheric  Administration. 

•  Performs  Software  Development ,  Operations  and 
Maintenance,  Information  Technology  Services, 
Engineering  Services  and  General  Site  services. 


©  2011  Lockheed  Martin  Corporation.  All  rights  reserved. 


Getting  Started 


•  Not  an  easy  thing  to  get  a  list  of  programs 

-  What  is  a  program?  (vs.  contract,  CLIN,  project) 

-  Who  would  have  the  most  current  list? 

•  And  what  would  it  include? 

•  Business  Area  (IS&GS)  Enterprise  Dashboard 

-  Centralized  portal  that  provides  weekly  insight  into 
IS&GS  programmatic  and  financial  performance  across 
programs  and  provides  a  consistent  consolidated  view 
of  performance  status 

-  Programs  over  a  certain  $  amount  required  to  report 
weekly 

•  Others  at  discretion  of  management 

•  Used  Enterprise  Dashboard  to  get  baseline  list  of  programs 

-  Already  provides  a  limiting  factor  ($  value)! 

-  Initial  list  included  132  programs 
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r  Sampling  Factors 

Sampling  factors  serve  to  identify  meaningful 
differences  in  the  conditions  under  which  work 
is  performed  in  the  organizational  unit.  Based  on  a 
thorough  understanding  of  the  organization,  the  lead 
appraiser  determines  the  sampling  factors  that  define 
different  clusters  of  process  implementation  for  the 
organization  unit.  Tiers  of  the  organization  chart  often 
provide  an  initial  view  of  these  potential  groupings.  The 
Method  Definition  Document  section  1.1.4  contains  a 


list  of  potential  sampling  factors  which  must  be 
evaluated.  In  addition,  the  lead  appraiser  seeks 
information  about  other  potential  sampling  factors. 


Standard  CMMI®  Appraisal  Method  for  Process  Improvement  (SCAMPISM)  A, 
Version  1.3:  Method  Definition  Document,  Appendix  F,  Page  212 


Sampling  Factors 

•  What  made  sense  as  a  differentiator  -  The  organization’s 
perspective: 

-  All  programs  are  required  to  follow  a  set  of  requirements,  tailored 
for  each  program  and  documented  in  a  compliance  matrix 

•  Regardless  of  Domain,  Geography,  Customer,  Size  or  Life 
Cycle 

•  Organizational  tailoring  was  done  for  Program  Type  (e.g., 
development,  services,  operations/maintenance) 

•  Requirements  were  specific  for  software,  systems,  hardware 

-  Therefore,  the  organization  felt  that  the  following  were  process 
differentiators: 

•  Discipline  (i.e.,  software,  systems,  hardware) 

•  Program  Type  (i.e.,  development,  services,  operations  & 
maintenance  -  O&M) 

•  Architecture  Methodology  (model-based  or  non-model  based) 

•  $  Value  (default  from  program  list  obtained) 

-  Provided  this  information  to  the  Lead  Appraiser  for  review  / 
comment 
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Sampling  Factors  (2) 


•  What  made  sense  as  a  differentiator  -  The  Lead  Appraiser’s 
perspective: 

-  Additional  information  requested  for: 

•  %  of  contracts  that  were  with  Defense  customers  (92%) 

•  Lines  of  Business  within  the  organization  (3  of  the  5  with 
development  or  O&M  effort  included  in  scope) 

-  Agreed  the  following  were  not  differentiators 

•  Domain,  Geography,  Customer  and  Life  Cycle 

•  Did  not  agree  that  size  was  not  a  differentiator 

-  Lead  Appraiser  felt  that  the  following  were  process 
differentiators: 

•  Discipline  (i.e.,  software,  systems,  hardware) 

•  Program  Type  (i.e.,  development,  services,  O&M) 

•  Architecture  Methodology  (model-based  or  non-model 
based) 

•  Size  (Full-Time  Equivalents) 
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Subgroups 

Sampling  factors  are  used  to  define  subgroups 
in  the  organizational  unit.  Subgroups  consist 
of  sets  of  basic  units  which  share  the 
attributes  identified  by  the  sampling  factors. 
Subgroups  are  defined  by  determining  all 
potential  combinations  for  each  value  of  the 
sampling  factors. 


Standard  CMMI®  Appraisal  Method  for  Process  Improvement  (SCAMPISM)  A, 

Version  1.3:  Method  Definition  Document,  Appendix  F,  Page  212 
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Sub-Groups 


First  Pass: 

-  19  Sub-groups  defined  based  on  Discipline,  Program 
Type,  and  Architecture  Methodology  (called  Project 
type) 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types 

D1 

H/W  O&M 

D2 

Integration 

D3 

O&M  Pass-thru,  Production 

D4 

PM  Only 

D5 

Production 

D6 

Research 

D7 

SW  &  HW  O&M 

DS 

SE  &SW  /  Development  /  non-model  based 

DS 

SE  &  SW &HW/ Development/ model  based 

DIO 

SE  &SW &HW/ Development/ non-model  based 

Dll 

SE  &  SW  &  HW  /  O&M 

D12 

Services 

D13 

SW  &  HW /  Development/  model  based 

D14 

SW  &  SE  /  Development/  model  based 

D15 

SW&  SE/  O&M 

D16 

SW&SE/Dev-  IDIQ 

D17 

SW/  Development  /  model  based 

DIE 

SW/  O&M 

D1S 

T&M 
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Sub-Groups  (2) 

*  Slice  &  Dice: 

-  Select  only  those  programs  that  were  Development 
or  O&M 

-  12  Sub-Groups 
remaining 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types 

D1 

H/W  O&M 

Of 

ii  i  auui  i 

DC 

r.  ..  ..  .i  i 

D4 

1  yi|  IP  u  Jj  Ji  ill  l  Jj.  |  L  J  yi  ^  ^  |  1^,1  1  1 

1  111  !w-  1  1  1  Jf 

DO 

1  1  ULIUL-llUI  1 

DO 

itcjcm  i.-i  i 

D7 

SW  &  HW  O&M 

DS 

SE  &  SW  /  Development  /  non-model  based 

DS 

SE  &SW &HW/ Development/ model  based 

DIG 

SE  &SW  &HW/ Development/ non-model  based 

Dll 

SE  &  SW  &  HW  /  O&M 

01 1 

Uc;  i  v  i  h_-t_  ji 

D13 

SW  &  HW /  Development/  model  based 

D14 

SW  &  SE  /  Development/ model  based 

D15 

SW&SE/O&M 

D16 

SW&  SE/Dev-  IDIQ 

D17 

SW/  Development:/  model  based 

DIB 

SW/ O&M 

D  1C 
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Sub-Groups  (3) 

*  Slice  &  Dice: 

-  Select  only  those  programs  that  were  multi- 
disciplined  (with  at  least  Systems  Engineering  for 
RD) 

-  6  Sub-Groups 
remaining 


©  2011  Lockheed  Martin  Corporation.  All  rights  reserved.  *  Found  out  this  wasn’t  a  real  “program”  -  no  task  orders  11 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types 

D1 

ii  nc£i  miui  i 

^JT.  »  ■  .ry  P  1,  .  ,.TV-  1  - : - ,  ■ 

JHH  II  U  J  J  LI  ll  r  IT  1  UUUULIUI  1 

DT 

1  1 1  1  >■_■■  Illy 

C.I  ■ 

1  1  ■_■  '_1  '_1  ■_  LI  1  1 

DO 

i  icjicdi  ■_■!  i 

SW  At  FIVif  U8W^™ 

DS 

SE  &SW  /  Development  /  non-model  based 

D9 

SE  &SW &HW/ Development/ model  based 

DIO 

SE  &SW  &HW/  Development/ non-model  based 

Dll 

SE  &SW&HW/0&M 

Dll 

v  ivcj 

J  11  l)  U.  II  1  U  U  f  ULUUU[JI..I^I.IL;  1  ■  1  %J  U  4  ■  HJ  LH  J  ■  'J 

D14 

SW  &  SE/  Development/  model  based 

D15 

SW&SE/O&M 

-i-  -m  ■ _ m _ 

Sub-Groups  (4) 


*  Slice  &  Dice: 


-  Select  only  those  development  programs  that  used 
model-based  architecture  design  (keeping  O&M) 


-  4  Sub-Groups 
remaining 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types 

pi 

Li  I\  hi  nP.ftii 

Cl 

II  ILCgl  dliui  1 

J-.  r,  •.  a  t-.  1  — .  II  .  - 

DC 

■■■■  ‘t.-*. i  i  i  raj*  Lm  4JJ  r  i  uuyuuui  i 

D4  ■ 

i  -  •■_■  ! 

r-.-  JL  

1  1  ■■  ■!  Rl  ■■  1  1 

DC 

ncjcai  '-i f 

DS 

SE  &SW  &HW/ Development/ model  based 

Dll 

SE  &SW&HW/ O&M 

D12 

■_M^  1  VI  '-■tC  _! 

D14 

SW  &  SE  /  Development/  model  based 

D15 

SW&  SE/  O&M 

Pi«3 _ 

-i-O  mm 

D» 
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Sub-Groups  (5) 

*  Reviewed  again  by  the  Lead  Appraiser: 

-  This  is  when  he  determined  that  Size  was  also  a 
factor! 

•  Start  all  over  again? 

•  Not  necessary  -  took  the 
remaining  sub-groups 
and  created  sub-sub- 
groups  based  on  size: 

•  <  50  Full  Time 
Equivalents  (FTEs) 

•  50 -100  FTEs 

•  >100  FTEs 

•  BUT  this  resulted  in  14 
programs  having  to  be 
sampled 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types  / Size 

D9 

SE  &  SW  Si  HW  /  Development/  model  based 

DS.l 

<50  FTE 

D9.2 

50  -  100  FTE 

D9.3 

>  100  FTE 

Dll 

SE  Si  SW  Si  HW  /  OSlM 

Dli.l 

<50  FTE 

Dll. 2 

50  -  100  FTE 

Dll. 3 

>  100  FTE 

D14 

SW  &  SE /  Development/ model  based 

D14.1 

<50  FTE 

D14.2 

50  -  100  FTE 

D14.3 

>  100  FTE 

D15 

SW  Si  SE/OSiM 

D15.1 

<50  FTE 

D15.2 

50  -  100  FTE 

D15.3 

>  100  FTE 
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Sub-Groups  (6) 

*  One  more  change  in  scope: 
-  Programs  with  >  50  FTEs 


*  Some  of  the  remaining 
sub-sub-groups  did 
not  have  any 
programs  that  fit 
within  that  sub-sub- 
group 


Subgroup 

Relevant  Sampling  Factors 

Disciplines  /  Effort  Types  /  Project  Types / Size 

D9 

SE  &  SW  &  HW  /  Development/  model  based 

D9.2 

50  -  100  FTE 

D9  .3 

>  100  FTE 

Dll 

SE  fit  SW  &  HW  /  OfitM 

D11.2 

50  -  100  FTE 

Dll. 3 

>  100  FTE 

D14 

SW  fit  SE /  Development/ model  based 

D14.2 

50  -  100  FTE 

D14.3 

>  100  FTE 

D15 

SW  fit  SE/OfitM 

D15.2 

50  -  100  FTE 

D15.3 

>  100  FTE 
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Sub-Groups  (7) 

*  Results 


Subgroup 

Relevant  Sampling  Factors 

Size  of  Sub-sub-group 

Size  *  #  Sub-sub-groups 

[Size  *  #  Sub-sub-groups)  /#  Programs 

#  Sampled 

Disciplines  /  Effort  Types  /  Project  Types  / Size 

D9 

SE  &  SW  Si  HW  /  Development  /  model  based 

D9.3 

>  100  FTE 

4 

20 

1.67 

2 

Dll 

SESi  SW  Si  HW/GSlM 

D11.3 

>  100  FTE 

3 

15 

1.25 

1 

D14 

SW  Si  SE/  Development/  model  based 

D14.2 

50  -  100  FTE 

3 

15 

1.25 

1 

D14.3 

>  100  FTE 

1 

5 

0.42 

1 

D15 

SW  Si  SE  /  OSlM 

D15.3 

>  100  FTE 

1 

5 

0.42 

1 

Total  Number  Programs 

12 

6 

Number  of  Sub-sub-groups 

5 

•  6  Programs  to  be  Sampled 

•  Evolved  Appraisal  Scope: 


-  Model  based  architecture  development  or  O&M  only, 
Multi-Disciplined  with  at  least  Systems  Engineering 
included,  for  programs  of  50  or  more  Full-Time- 
Equivalent  people 
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Groups  of  people  who  perform  work  that 
enables  or  supports  the  work  which  is 
ultimately  visible  to  the  customers  of  the 
organization. 


Standard  CMMI®  Appraisal  Method  for  Process  Improvement  (SCAMPISM)  A, 
Version  1.3:  Method  Definition  Document,  Appendix  F,  Page  212 


Support  Functions 


*  Looked  across  the  organization’s  structure  to  see  what 
could  be  defined  as  a  “Support  Function” 

-  Quality  -  independent  from  the  programs 

-  Subcontracts  -  all  follow  Corporate  guidelines  and 
procedures 

-  Organizational  groups 

*  Received  additional  questions  from  the  Lead  Appraisers: 

-  Provide  information  on  reporting  and  reviews  for  Quality 
activities 

-  What  type  of  supplier  agreements  exist 

*  Worked  with  the  organization  to  obtain  additional 
information 
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Support  Functions  (2) 


•  Quality  was  defined  to  be  a  Support  Function 

-  Different  reporting  structure  from  Program  Management 

-  Monthly  Operating  Review  by  Quality  manager 

•  Covers  key  accomplishments,  disappointments,  past  30 
days  activity,  future  30  day  activity,  audits,  corrective 
actions,  prevention  measures,  etc. 

-  Quality  Plans  had  already  been  established  on  the  programs 

•  Quality  manager  participates  in  yearly  program  reviews, 
reviewing  the  Quality  Plans  to  ensure  they  are  up-to-date 
and  are  being  followed 

•  Although  this  could  have  limited  evidence  collection,  all  the 
programs  decided  to  provide  PPQA  evidence  as  part  of  their 
preparation  for  the  SCAMPI  A! 
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Support  Function  (3) 


*  Subcontracts  was  not  defined  to  be  a  Support 
Function 

-  Corporate  terms  were  not  the  same  as  model  terms 

•  Difficult  to  explain  to  participants 

•  Difficult  to  explain  to  Lead  Appraiser 

-  Different  levels  and  combinations  of  supplier 
activities 

•  From  standard  COTS  purchases  to  full 
subcontract  management 

*  Could  have  pushed  to  make  this  a  support  function 
because  of  the  consistency  across  the  organization 

-  But  it  was  “too  hard”! 
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End  Result  -  So  Far 


•  Sampling  Factors  have  been  identified 

-  Modified  the  original  organizational  scope 

-  Several  iterations  with  Lead  Appraiser  required 

•  To  understand  model  perspective 

•  To  understand  organization 

•  Programs  have  been  identified 

-  5  programs  to  provide  full  PHD  evidence  (except  PPQA) 

-  2  programs  to  provide  2  or  3  PAs 

-  Quality  organization  to  provide  PPQA  evidence  from  one  of 
the  programs 

•  Data  Coverage  Plan  completed 

-  Team  review  of  evidence  completed 
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End  Result  -  So  Far  (2) 


•  Would  it  have  been  the  same  under  VI  .2? 

-  It  depends 

-  In  the  past  (under  VI  .1  and  VI  .2): 

•  Identified  minimum  of  3  programs  that  provided  45  - 
50%  of  sales  and  people 

-  Typically,  the  biggest  programs 

•  In  addition,  would  identify  5-10  programs  that  would 
prepare  evidence  to  be  “sampled”  if  the  appraisal 
team  wanted  additional  institutionalization  evidence 

•  8-13  programs  preparing  full  sets  of  evidence 

-  Probably  would  have  identified  some  of  the  same 
programs 

•  But  only  5  programs  providing  full  sets  of  evidence 
and  2  more  providing  2-3  process  areas 
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Lessons  Learned 


•  SCAMPI  VI  .3  takes  more  effort  in  up-front  planning 

•  SCAMPI  VI  .3  required  earlier  involvement  by  Lead 
Appraiser 

-  Lead  needs  to  really  understand  the  organization 

-  Lead  needs  to  be  responsive  to  questions  /  issues  raised 
by  organization  so  that  the  “slicing  and  dicing” 
proceeds  quickly 

-  Organization  needs  to  be  responsive  to  questions  from 
the  Lead 

•  This  organization’s  programs  are  risk-averse 

-  Even  when  they  don’t  have  to  provide  evidence,  they  do 
anyway 

•  Just  in  case 

•  So  they  won’t  have  to  do  something  at  the  last  minute 
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Contact  Information 


•  Lynn  Penn:  marv.lvnn.penn@lmco.com 

•  Dorna  Witkowski:  dorna.witkowski@lmco.com 
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Questions?? 
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Agenda 

■  IDS  History 

■  Cost  Effective  SCAM  Pi’s  -  a  MUST 

■  Bargain  Hunting 

■  Key  Shopping  Tips 

■  Summary 


Raytheon 


Is  it  possible  to  reduce  SCAMPI  costs? 

■  The  considerable  cost  of  preparing  for  and  conducting  a  SCAMPI  Class  A 
appraisal  can  reduce  the  value  of  a  CMMI  initiative  as  well  as  diverting 
funds  away  from  improvement  activities. 

■  This  presentation  will  share  how  Raytheon  Integrated  Defense  Systems 
(IDS)  was  able  to  significantly  reduce  cost  and  cycle  time  for  preparing  for 
and  conducting  a  201 1  CMMI-DEV  VI  .3  Maturity  Level  5  SCAMPI  A  as 
compared  to  the  similar  scope  SCAMPI  A  conducted  in  2008. 

■  The  strategy  used  balanced  results  with  an  acceptable  level  of  risk. 

■  We  hope  you  can  take  some  of  our  lessons  learned  and  apply  them  to 
your  organization. 
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IDS  CMMI  Appraisal  History 


First  CMMI 
SCAMPI 
with  SEI 
Pilot  2000 


CMMI  ML  3 
SE/SW 
2003 


CMMI  ML  4 
SE/SW 
CMMI  ML  3 
HW 
2005 


CMMI  ML  5 
SE/SW/HW 
2008 


CMMI-SVC  ML  3  CMMI  ML  5 
WLE  SE/SW/HW 

2010  2011 

(Re-Appraisal) 
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IDS  CMMI  Appraisal  History 


First  CMMI 
SCAMPI 
with  SEI 
Pilot  2000 


CMMI  ML  3 
SE/SW 
2003 


CMMI  ML  4 
SE/SW 
CMMI  ML  3 
HW 
2005 


CMMI  ML  5 
SE/SW/HW 
2008 


CMMI-SVC  ML  3 
WLE 
2010 


CMMI  ML  5 
SE/SW/HW 
2011 

(Re-Appraisal) 
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Doesn’t  Everyone  Want  a  Bargain? 


A  Bargain! 


Bargain  (noun):  ba:(r)  gin  (Macmillan  Dictionary) 

1 .  something  you  buy  that  costs  much  less  than  normal 

You  should  be  able  to  pick  up  a  few  good  bargains. 

a.  a  lower  than  usual  price 

Twenty  pounds  is  a  real  bargain! 
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How  do  YOU  Shop  for  a  Bargain... 


Raytheon 
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How  do  YOU  Shop  for  a  Bargain... 


Special  Tips  for  Getting  the  Best  Bargains  on  Black  Friday. . . 


Raytheon 


Tips  for  Bargain  Hunting... 
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#1  Shopping  List 

What  do  we  have  to  get ...  It’s  all  in  Planning! 


Problem:  Evidence  had  been  collected  from  multiple  disciplines  for  every  Process  Area  and 
was  Evidence  Collector  Driven 

Goal:  To  develop  an  Efficient  Data  Collection  Strategy 

■  Established  a  Plan  as  to  who  would  supply  what  evidence 

-  Appraise  at  the  program  level  vs.  by  individual  disciplines 

-  One  Evidence  Collector  per  Program 

■  Developed  Precision  PI  ID’s  -  “Operational  Definitions” 

-  Example  Evidence  very  detailed 

-  Common  Artifacts  Identified 

-  Identified  Multi  Purpose  Evidence  “Threads”  for  ease  of  collection  and  appraising 

■  Established  Common  File  structure  for  Evidence  Repository 


If  you  can't  explain  it  simply,  you  don't  understand  it  well  enough. 


Raytheon 
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#2 


Deals 


What’s  unique  ...  It’s  all  in  Understanding  the  Business  Needs  and 
Tailoring  to  suit! 

Problem:  Appraisals  were  considered  a  Major  Event  requiring  many  Meetings  and 
Communications 

Goal:  To  make  Appraisals  Less  Invasive  to  Programs 

■  Eliminated  Participant  Briefing  &  Opening  Briefing  (3  Briefings  merged  to  1) 

-  Used  existing  Project  Briefing  Sessions,  combining  duplicate  information 

■  Eliminated  two  “Formal”  Preliminary  Findings  Briefings 

-  Performed  alternative  validation  of  Preliminary  Findings  -  via  E-mail  with  acknowledgement  of  Findings 

■  Eliminated  the  appraisal  “hype” 

-  Reduced  Communications  Costs  -  We  did  not  want  to  “alert”  the  organization 

-  An  appraisal  event  should  not  be  a  stimulus  to  perform  differently! 

•  “This  is  the  way  we  do  business” 

-  Stoplight  status  stayed  within  “Closed  doors”  (vs.  Fix  that  RED!) 

Breaking  with  Traditional  Thinking 

Raytheon 
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#3  Coupons 

How  much  can  we  get  for  our  money  ...  It’s  all  in  spending  $  wisely! 


Problem:  Too  much  Time  and  Money  was  spent  on  Polishing  the  Evidence  and  conducting 
Multiple  Appraisal  Events 

Goal:  Ensure  Appraisals  do  not  consume  majority  of  Funding... Budget  spread  to  Sustainment, 
Institutionalization  and  Improvements 


■  Focused  on  Sustainment  Activities 

-  Updated  Quality  Audits  to  find  sustainment  issues 

-  Used  a  Maturity  Index  Tool  to  ensure  no  process  regression 


■  Reduced  Evidence  review  time  Pre-Onsite  and  Onsite  Activities 


-  Common  Artifacts  and  Threads  told  the  same  story 

■  Eliminated  Class  C  and  B  appraisals  -  Were  not  looking  for  an  appraisal  with  “No  findings” 

-  Address  weaknesses  after  the  SCAMPISM  rather  than  between  events 

■  External  “seasoned”  team  members  reviewed  high  risk  evidence 

■  Level  Set  the  Appraisal  Team  by  phone  a  few  weeks  prior  to  the  appraisal 


More  Bang  for  the  Buck! 


Raytheon 
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#4  Buy  1  Get  1  Free 

What  can  we  get  for  free  ....  It’s  all  in  the  Re-Use! 


Problem:  Multiple  copies  of  the  same  evidence  collected  by  various  evidence  collectors  -  too 
much  evidence,  too  time  consuming  to  review/interpret 

Goal:  To  collect  the  Minimum  amount  evidence  -  for  the  Maximum  amount  of  coverage 


■  Developed  a  list  of  “common”  artifacts 

-  30/40  key  documents  provided  70%  coverage  ... 

-  Reduced  Evidence  Collection  Rework 

■  Established  a  Common  File  Structure  for  Evidence 

-  Ensured  ease  of  evidence  identification  by  collector  and  reviewer 

■  Created  one  PHD  per  Program  being  appraised  (vs  PIIDs  by  PA) 

-  Facilitated  the  copy  of  common  links  rather  than  having  to  relink 

■  Identified  One  Evidence  Collector  per  Program 


-  Evidence  collector  knew  what  was  already  collected  and  could  re-use  across  Process  Areas 


Read  Once....  Write  Many 


Raytheon 


#5  Comparison  Shop 


Who  has  the  best  ...It’s  all  about  Lessons  Learned! 

Problem:  Preparing  for  and  Conducting  Appraisals  the  same  as  we  always  had  because 
“that’s  how  we  do  it” 

Goal:  Utilize  Lessons  Learned  to  get  a  Lean  appraisal  process  while 
Integrity  and  Accurate  Appraisal  findings. 

■  Write  Once...  Read  Many  for  Evidence  Collection 

■  Logistics 

-  “Dry  Run”  in  New  Conference  Area 

-  Ensured  two  site  coordinators  were  available  and  at  least  one  was 
present  to  assist  the  appraisal  team 

■  Business  Units  typically  staff  50%  of  their  team  members  from  outside 

-  Enables  sharing  of  best  practices  across  the  entire  Raytheon  -  leverage  across  and  take  the  best  of 
the  best 

■  Utilized  local  experienced  appraisal  team  members  for  evidence  review 

Work  Smarter,  Not  Harder 


Raytheon 


Impulse  Purchase 

Seeing  something  you  can’t  live  without...  It’s  all  about  Best  Practices! 


■  Raytheon  shares  among  6  Business  Units  (BU) 

-  Engineering  process  Group  (EPG)  Workshop 

-  Constantly  look  for  improvements  (Feeding  ML  5) 

-  Hi -MAT 


-  Worked  with  Programs  in  their  own  language  -  “not  in 
determine  actual  process  execution 

•  Models  that  make  sense  -  “Model  Hunting” 

•  Programs  were  performing  “High  Maturity” 

■  Maturity  Index  Tool 

-  Adopted  from  another  BU 

•  Altered  to  fit  our  process  structure 

•  Modified  to  incorporate  Services  Model 

■  Appraisal  Team  Members 

-  Participate  on  other  BU  Appraisals 

-  Bring  members  from  other  BU’s  to  participate  on  ours 

■  Lead  Appraiser 


CM  Ml  terms”  to 


-  Periodic  Site  Visits  keeps  Lead  involved  -  no  surprises 


Raytheon 


Our  ML5  Re-appraisal . a  Bargain! 


March  2011 

CMMI®  ML5 
SCAMPISM 


-50%  less  overall  cost  (Prep  &  Conduct) 

-  2.5  weeks  on-site  vs.  4  weeks 

-  8  months  ahead  of  schedule 

-  Minimal  impact  to  programs  -  Already  sustaining 

Raytheon 
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Questions? 


Raytheon 


Contact  Information 


Mary  Ellen  Christopher 


Mary_Ellen_S_Christopher@raytheon.com 

978.858.9013 

Raytheon  Integrated  Defense  Systems 
Tewksbury,  MA  01876 


Jayne  Perkins 


Jayne_A_Perkins@raytheon.com 

978.858.4506 

Raytheon  Integrated  Defense  Systems 
Tewksbury,  MA  01876 
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Appraisals  and  CMMI  Gotchas 

Lessons  in  CMMI  Use 
and  Appraisal  Preparation 


Neil  Potter 
The  Process  Group 
help@processgroup.com 


Referenced  articles  are  at  www.processgroup.com/newsletter.html 
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Agenda  -  Part  1 


*  Introduction 

*  CMMI  Premise 

*  Documentation 

*  Configuration  Management 

*  Measurement  and  Analysis 

*  Project  Planning 

*  Project  Monitoring  and  Control 
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*  Integrated  Project  Management 

*  Training 

*  Equal-weighted  Process  Area  practices? 

*  Appraisal  Preparation  -  PIIDing 

*  Appraisal  Interview  Preparation 
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CMMI  HAZARDS! 

Introduction 


Using  CMMI  or  preparing  for  an  appraisal? 

-  Avoid  the  hazard  of  creating  a  paper 
factory,  instead  focus  on  organizational 
results 

-  Avoid  putting  the  emphasis  on  the  less 
important  issues 

»  e.g.,  policy  recital,  training  records, 
emails  that  say  “We  assigned  this  to 
Fred” 

-  Spend  your  time  making  things  better,  not 
on  a  rote  exercise 

-  Know  some  common  blind  spots 
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CMMI  Premise 

*  CMMI  practices  can: 

-  Reduce  project  risk 

-  Reduce  rework  and  costs 

-Improve  output  quality  and  predictability 

-Improve  productivity  through  process  improvement  and 
process  reuse 

•  CMMI: 

-Can  be  used  to  diagnose  current  state 

-  Provides  an  example  roadmap  forward 

»  Management/project,  engineering/organization, 
statistics/prediction,  variation/mean 


©  Copyright  2007-2011  The  Process  Group.  All  rights  reserved. 


www. processgroup.com 


Version  1.5,  45min 


5 


THE 


PROCESS 

GROUP 


Hazard:  Drowning  in  Documentation 

•  Easy  to  fall  into  the  trap  of  the  paper 
factory 

-  We  are  developers,  so  we  develop! 

-  What  we  really  need  is  guidance  for  our 
jobs 

»  Capture  best  organization 
engineering  and  management 
practices 

»  Not  necessarily  repeat  every  book 
known  to  mankind! 

•  What  problem  are  we  trying  to  solve? 

-  Make  engineering  easier,  quicker,  less 
hassle  -  NOT  MORE 

[Newsletter  “documentation”] 
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Configuration  Management  (CM) 

Hazard:  over-simplification 

•  CM  looks  pretty  straight  forward,  once  people  start  to  understand 
the  discipline 

•  Don’t  avoid  Configuration  audits  -  make  them  useful  [SP  3.2] 

-  Use  physical  audits  to  help  ensure  that  products  are  released 
correctly,  e.g., 

»  Verify  differences  between  source  and  release  =  change  list 
»  Compare  checksum  value  between  source  and  release 

•  What  problem(s)  are  we  trying  to  solve? 

-  Producing  the  right  stuff  and  getting  it  to  the  customer 

-  Keeping  track  of  our  stuff,  protecting  ourselves  from  loss 


SP  3.2:  Perform  configuration  audits  to  maintain  integrity  of  the  configuration  baselines. 
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Measurement  and  Analysis  (MA) 

Hazard:  skip  parts  or  overkill 


•  Organizations  often  have  metrics  but  entirely  skip  the  first  half  of  this 
Process  Area: 

-  Defining:  objectives,  metrics,  analysis,  reporting,  information  storage 

•  Or  take  the  other  extreme  and  overdo  measurement  and  goal 
definitions 

-  34  objectives,  a  procedure  for  documenting  objectives,  82  core  metrics 

•  Need  a  good  balance  for: 

-  Spending  enough  time  to  arrive  at  appropriate  goals 

-  Specifying  what  measures  are  needed 

-  Clarifying  how  they  will  be  analyzed  and  stored 

•  What  problem  are  we  trying  to  solve? 

-  Knowing  why  we  are  measuring  in  order  to  get  the  most  value  out  of  it 
and  not  waste  time  on  useless  metrics 


[Newsletter  “measurement”] 
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GP  2. 8/3.2  and  Over-simplified  MA 

Hazard:  I  measured  it  because  CMMI  SAID  I  HAD  TO! 


MA  comprises  of  only  7  PA  measures,  and  GP  2.8  and 
3.2  are  academic 

-  What  is  it  telling  you? 

What  problem  are  we  trying  to  solve? 

-  GP  2.8  (on  each  PA)  -  How’s  it  going  this  time? 

-  GP  3.2  (on  each  PA)  -  Are  the  PA  related  processes  as 
implemented  meeting  our  needs,  getting  better  or 
worse? 

-  MA  should  help  you  run  your  business,  not  just  CMMI! 


GP  2.8:  Monitor  and  control  the  process  against  the  plan  for  performing  the  process  and  take  appropriate  corrective  action. 

GP  3.2:  Collect  process-related  experiences  derived  from  planning  and  performing  the  process  to  support  the  future  use  and 
improvement  of  the  organization’s  processes  and  process  assets. 
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Project  Planning  (PP) 

Hazard:  skimping  on  size  estimation  and  risk  management 


Many  people  either  skip  size,  or  don’t  spend  enough  time  finding  a 
good  use  for  size  or  attribute  estimation  [SP  1.2] 


-  “My  project  size  is  2,000  hours” 

-  “I  estimate  Lines  of  Code,  but  track  effort” 

Others  underutilize  risk  at  the  project  level  [SP  2.2] 

-  Risks  should  come  from  the  team,  not  just  the  manager 

-  Risks  should  be  more  than  boilerplate  “We  might  not  have 
resources” 

-  Risks  should  be  made  very  visible  to  customers  +  management 
What  problem  are  we  trying  to  solve? 

-  Clarifying  how  big  the  project  is 

-  Understanding  what  can  really  go  wrong 

-  Thinking  through  potential  issues  ahead,  while  there  is  time  to 
react  /  recover 


SP  1.2  Establish  and  maintain  estimates  of  work  product  and  task  attributes. 
SP  2.2  Identify  and  analyze  project  risks. 


[Newsletter  “attributes”] 
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Project  Monitoring  and  Control  (PMC) 

Hazard:  missing  valuable  information  that  could  save  the  day 

•  No  useful  way  to  track  actual  work  progress  [SP  1.1],  e.g., 

-  Actual  work  effort  (labor) 

-  Actual  amount  of  work  accomplished  (size) 

•  What  problem  are  we  trying  to  solve? 

-  Use  data  to  determine  if  current  resource  expenditure  (hours  or 
money)  can  be  sustained 

-  Know  the  volume  of  work  and  how  much  each  project  actually 
costs 


»  How  much  we  lost  this  time,  or  how  much  future  projects  might  cost 

-  Proactively  manage  and  identify  re-planning  points  while  there  is 
time  to  recover 

»  Identifying  large  changes  in  effort  or  size 


SP  1 .1  Monitor  actual  values  of  project  planning  parameters  against  the  project  plan. 


[Newsletter  “attributes”] 
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Integrated  Project  Management  (IPM) 

Hazard:  not  having  proactive  visibility 

*  Not  use  thresholds  to  trigger  corrective  action  [SP  1 .5] 

-  At  Level  3,  corrective  action  and  escalation  are  more  objective  (“We 
are  10%  behind”)  than  emotional  (“I  think  things  will  speed  up”) 

-  Organizational  and  project  knowledge  are  used  to  establish 
thresholds 

*  Process  tailoring  not  based  on  organizational  learning  [SP  1.1] 

-  Level  3  is  often  interpreted  as  “Processes  are  standardized  across 
all  projects,”  rather  than  “Standard  processes  are  tailored  for  each 
project” 

*  What  problem  are  we  trying  to  solve? 

-  We  have  MEANINGFUL  data,  let’s  really  use  it! 

-  Have  organizational  wisdom  available  and  used 


SP  1 .5  Manage  the  project  using  the  project  plan,  other  plans  that  affect  the  project,  and  the  project’s  defined  process. 
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Integrated  Project  Management  (IPM) 
Without  Historical  Data? 

Hazard:  databases  full  of  data  are  not  enough! 

*  Organizational  Process  Definition  (OPD) 
and  IPM  not  well  understood 

-  OPD  sets  up  a  Process  Asset  Library  and 
measurement  repository  for  use  by 
projects  (IPM) 

-  Not  all  Lead  appraisers  know  or 
communicate  this 

*  What  problem  are  we  trying  to  solve? 

-  Run  projects  based  on  historical  and 
current  data 
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Do  Software  Engineers  Need  Training? 

Hazard:  trivial  training 

•  Project  Planning  (SP  2.5) 

-  Make  sure  you  have  the  skills  for  THIS 
project 

•  Organizational  Training 

-  Make  sure  you  have  the  skills  for  current 
work,  and  work  to  come 

•  What  problem  are  we  trying  to  solve? 

-  Engineers  and  managers  don’t  have  the 
skills  to  perform  their  roles  correctly  (as 
per  process  definition)  and/or  efficiently 

-  Prevent  mistakes  due  to  lack  of  skills 
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Equal-weighted  Process  Area  practices? 

Hazard:  each  process  area  practice  is  treated  as  EQUAL 

•  Each  CMMI  practice  should  not  necessarily  be  equally 
weighted  during  implementation.  Example: 

-  Policy  vs.  estimating  effort  or  risk 
-Training  records  vs.  performing  validation 

•  The  correct  weighting  can  be  given  when  you: 

-  Focus  on  what  you  are  trying  to  accomplish  (real  jobs) 

-  Use  the  CMMI  and  its  components  to  improve 

-  Fix  real  problems 

•  What  problem  are  we  trying  to  solve? 

-  Real  world,  day-to-day  work  gets  better  (easier,  faster,  higher 
quality,  less  stress,  less  busy-work,  less  rework,  less  risk) 
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Appraisal  Preparation  -  PIIDing 

Hazard:  creating  documents  to  please  the  appraiser 

•  As  an  appraisal  date  approaches,  people  find  themselves 
focused  on  providing  required  appraisal  evidence: 

-  A  lot  of  time  can  be  wasted  chasing  down  documents 

-  When  practices  are  institutionalized  correctly,  the  evidence 
needed  already  exists 

*  What  problem  are  we  trying  to  solve? 

-  Evidence  should  never  be  created  to  please  an  appraiser 

-  Artifacts  examined  should  be  the  real  work  of  the  organization 

-  For  example,  evidence  of  responsibilities  could  be  an 
organization  chart  or  a  schedule  with  assignments 

*Practice  Implementation  Indicator 
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Appraisal  Interview  Preparation 

Hazard:  wasting  time  rehearsing 

•  Some  people  prepare  using  mock  interviews 

-  Appraisals  should  be  about  how  you  DO  YOUR  REAL 
work 

-  Interview  practice  might  make  folks  feel  more 
comfortable,  but  this  can: 

»  Induce  stress  over  remembering  to  say  the  right 
answers 

»  Focus  your  people  on  CMMI  terms  and  rote 
answers 

•  What  problem  are  we  trying  to  solve? 

-  Time  to  practice  for  an  appraisal  takes  away  from 
getting  real  work  done 

-  Participants  should  be  able  to  answer  the  questions 
because  the  answers  describe  how  they  do  their  jobs 
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Using  Lessons  Learned  from  Medical 
Checklists  to  Simplify  CMMI  Processes 


The  Process  Group 
www.processgroup.com 
neil@processgroup.com 


Neil  Potter 


SM  CMMI  is  a  service  mark  of  Carnegie  Mellon  University. 
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Management  (REQM)  Process 
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Need 
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Background 

World  Health  Organization  Goal:  Improve  surgical 
procedures  worldwide*. 

A  medical  checklist  was  created  based  on  aviation 
checklists. 

Premise:  a  simple  checklist  can  ensure  that  critical  steps 
have  not  been  overlooked,  either  due  to  haste, 
forgetfulness  or  inexperience. 


•  Measurements  were  collected  from  surgeries  performed 
around  the  world,  before  and  after  the  checklist.  Results: 

-  Major  complications  down  by  36% 

-  Infections  down  by  -50% 

-  Patients  returned  to  surgery  because  of  problems  down  by  25% 

-  Harm  suffered  from  surgery  (over  4,000  patients)  down  by  150 

-  27  fewer  deaths  (47%  drop)  caused  from  surgical  complications 


*Atul  Gawande,  associate  professor  of  surgery  at  Harvard  Medical  School 

Checklist  Manifesto:  How  to  Get  Things  Right 

Version  1  4 
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(with  at  least  nurse  and  anaesthetist) 


(with  nurse,  anaesthetist  and  surgeon) 


(with  nurse,  anaesthetist  and  surgeon) 


Has  the  patient  confirmed  his/her  identity, 
site,  procedure,  and  consent? 

□  Yes 


Is  the  site  marked? 

□  Yes 

□  Mot  applicable 


¥ 


Is  the  anaesthesia  machine  and  medication 
check  complete? 

□  Yes 

Is  the  pulse  oximeter  on  the  patient  and 
functioning? 

□  Yes 

Does  the  patient  have  a: 

Known  allergy? 

□  Md 
Q  Yes 

Difficult  airway  or  aspiration  risk? 

□  Mo 

□  Yes,  and  equipment/assistance  available 

Risk  of  >500ml  blood  loss  (7ml/kg  in  children)? 

□  Mo 

□  Yes,  and  two  IVs/central  access  and  fluids 
planned 


□  Confirm  all  team  members  have 
introduced  themselves  by  name  and  role. 

□  Confirm  the  patient's  name,  procedure, 
and  where  the  incision  wilt  be  made. 

Has  antibiotic  prophylaxis  been  given  within 
the  last  SO  minutes? 

□  Yes 

□  Mot  applicable 


Anticipated  Critical  Events 
To  Surgeon: 

□  What  are  the  critical  or  non-routine  steps? 

□  How  long  will  the  case  take? 

□  What  is  the  anticipated  blood  loss? 

To  Anaesthetist: 

□  Are  there  any  patient-specific  concerns? 

To  Nursing  Team: 

□  Has  sterility  (including  indicator  results) 
been  confirmed? 

□  Are  there  equipment  issues  or  any  concerns? 

Is  essential  imaging  displayed? 

D  Yes 

□  Not  applicable 


¥ 


Nurse  Verbally  Confirms: 

□  The  name  of  the  procedure 

□  Completion  of  instrument,  sponge  and  needle 
counts 

□  Specimen  labelling  (read  specimen  labels  aloud, 
including  patient  name) 

□  Whether  there  are  any  equipment  problems  to  be 
addressed 

To  Surgeon,  Anaesthetist  and  Nurse: 

□  What  are  the  key  concerns  for  recovery  and 
management  of  tills  patient? 
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Matching  Process  Complexity  with  Need 

•  The  time  needed  to  write  a  process  is  usually  a  lot  less  than  the 
time  spent  using  it.  E.g., 

-  A  project  planning  process  of  1-2  pages  may  take  a  2  days  to  develop  and  then 
be  used  numerous  times. 

-  A  spreadsheet  for  risk  management  may  take  V2  day  to  develop  and  manage 
numerous  risks  over  many  years. 

-  The  benefit  of  using  it  outweighs  the  cost  of  developing  it. 


•  Places  where  a  small  process  might  be  adequate: 


-  SVC:  REQM,  PP,  PMC,  CM,  MA,  OPF,  OPD,  OT. 


-  DEV:  OPF,  OPD,  PI  (small  DEV  projects),  OT 

-  ....  process  includes  all  Generic  Practices 
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Guidelines  for  Creating  Checklists  -i 

•  Two  main  styles  of  checklists: 

-  “Do-Confirm”  -  verify  critical  steps 

-  “Read-Do”  -  perform  given  specific  situations 

•  Select  pause  points  in  work  flow  where  the 
completion  of  critical  steps  can  be  verified. 

•  Condense  the  checklist  onto  one  page  and  use 
single  bullet  point  sentences. 

•  Checklist  items  are  critical  (high-risk)  and  are  not 
covered  elsewhere. 

•  Run  the  checklist  verbally  with  the  team  to 
ensure  that  anyone  that  has  an  issue  can  speak 
up. 

•  Revise  the  checklist  numerous  times  until  it  is 
able  to  quickly  detect  serious  problems. 


Read-Do 

CM  Process 

1 .  List  Configuration 
Items 

x,  y,  z 

2.  Establish  File  Naming 

Convention 

File-x<n>.docx 

3.  Establish  Baseline 
File  Structure 
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Guidelines  for  Creating  Checklists  -2 


•  Move  implementation  details  as  help  text  or  training. 

•  Treat  the  process  as  “day-to-day  usage,”  not  beginner. 

-  This  is  where  the  majority  of  time  will  be  spent. 

•  Example  scenario: 

-  Service  requirements  don't  change  much  (or  at  all)  over  time. 

-  REQM  is  used  1  day  per  year  to  manage  the  change. 


Service  delivery 


Service 

requirements 

update 
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Example  Service  Requirements 
Management  (REQM)  Process  -i 

Purpose:  A  checklist  used  to  understand,  confirm  and  manage  changes  to  service 
requirements. 

Policy:  All  service  changes  are  managed  using  this  checklist  [gp2.1] 

Do 

□  Plan  the  requirements  definition/review  event  [gp2.2]: 

•  Date: _ 

•  Time  /  resources  needed:  (gp2.3) _ 

•  Responsibility:  [gp2.4] _ 

•  Stakeholders  [spl. 2,  gp2. 7]: 

❖  Role  =  Agree  to  services:  <Name>.  Commitment _ 

❖  Role  =  Provide  expertise:  <Name>.  Commitment _ 

❖  Role  =  Team  member  1:  <Name>.  Commitment _ 

❖  Role  =  Senior  manager  approval:  [gp2.10]  <Name>.  Commitment _ 
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Example  Service  Requirements 
Management  Process  -2 

□  Discuss  new  and  changed  service  requirements  with  stakeholders  to  clarify 
understanding:  [spl.1,  1.3,  1.5] 

•  Review  current  service  requirements 

•  Review  proposed  changes  to  service  requirements 

•  Human  resources  needed  to  implement  change: _ 

•  New  materials/consumables/computers  needed  to  implement  change: _ 

•  Current  commitments  and  deadlines  impacted: _ 

•  Added  risks  and  mitigation  actions: _ 

•  Record  stakeholder  commitments  next  to  name  [spl  .2] 

□  Record  major  issues/actions 

□  Update  traceability  mapping  [spl. 4] 

•  Label  service  requirement  1  thru  N 

•  List  impacted  deliverables  and  documents  for  each  requirement 

•  State  test  method  (e.g.,  peer  review,  test  case,  pilot)  for  each  service  requirement 

□  Save  this  document  as  service-roles-vN.doc  on  X  drive  with  change  history 
comments  [gp2.6] 
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Check 

□  Training  has  been  provided  to  perform  the  steps  above?  [gp2.5] 

•  If  not,  training  date  /  time  /  who _ 

□  All  process  steps  above  have  been  performed?  [gp2.8] 

•  Corrective  actions  needed/taken? _ 

□  Objective/independent  check  done  [gp2.9]: 

•  Auditor  name: _ 

•  Audit  date: _ 

•  Pass  /  fail?: _ 

•  If  fail,  corrective  actions  needed: _ 

□  Senior  management  aware  of  this  service  requirements  event,  results,  issues? 
[gp2.10] 

•  Comments: 
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Development 


Drafting 


Validation 


□  Do  you  have  clear,  concise 
objectives  for  your  checklist?^™ 


Is 

□ 


each  item: 


A  critical  safety  step  and  in  great 
danger  of  being  missed? 


□  Not  adequately  checked  by  other 
mechanisms? 


□  Actionable,  with  a  specific 
response  required  for  each  item? 

□  Designed  to  be  read  aloud  as  a 
verbal  check? 


□  One  that  can  be  affected  by  the 
use  of  a  checklist? 


Have  you  considered: 


□  Adding  items  that  will  improve 
communication  among  team 
members? 


□ 


Involving  all  members  of  the  team 
in  the  checklist  creation  process? 


Does  the  Checklist: 

□  Utilize  natural  breaks  in  workflow 
(pause  points)? 

□  Use  simple  sentence  structure  and 
basic  language? 

□  Have  a  title  that  reflects  its 
objectives? 

□  Have  a  simple,  uncluttered,  and 
logical  format? 

Fit  on  one  page? 


□ 

□ 


4 


Minimize  the  use  of  color? 


Is  the  font: 


□  Sans  serif? 

□  Upper  and  lower  case  text? 

□  Large  enough  to  be  read  easily? 

□  Dark  on  a  light  background? 


□  Are  there  fewer  than  1 0  items  per 
pause  point? 


U  Is  the  date  of  creation  (or  revision) 
clearly  marked? 


Have  you: 

□  Trialed  the  checklist  with  front  line 
users  (either  in  a  real  or  simulated 
situation)? 

□  Modified  the  checklist  in  response 
to  repeated  trials? 

Does  the  checklist: 

□  Fit  the  flow  of  work? 

□  Detect  errors  at  a  time  when  they 
can  still  be  corrected? 

□  Can  the  checklist  be  completed  in 
a  reasonably  brief  period  of  time? 

□  Have  you  made  plans  for  future 
review  and  revision  of  the 
checklist? 


Source:  www.projectcheck.org 
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Summary 

•  Processes  don’t  have  to  be  voluminous  to  be  “complete.” 

•  A  checklist  is  adequate  for  some  Process  Areas  (and 
processes). 

•  Consider  splitting  processes  into  2  parts: 

-  a)  A  checklist  for  essential  steps  -  day-to-day  usage. 

-  b)  Separate  training/details/explanation. 

•  Use  “Do-Confirm,”  or  “Read-Do”  style. 

•  Refine  checklist  until: 

-  It  achieves  the  desired  result. 

-  It  is  able  to  quickly  detect  serious  problems. 
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•  PP/WP: 

•  PMC/WMC: 

•  CM: 

•  REQM: 

•  MA: 

•  OPF: 

•  OPD: 

•  OT: 

•  PI: 


Project  /  Work  Planning 
Project  /  Work  Monitoring  &  Control 
Configuration  Management 
Requirements  Management 
Measurement  Analysis 
Organizational  Process  Focus 
Organizational  Process  Development 
Organizational  Training 
Product  Integration 
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REQM  Practice  definition  from  CMMI 

•  SP  1 .1  Develop  an  understanding  with  the  requirements  providers  on  the  meaning  of  the 
requirements. 

•  SP  1 .2  Obtain  commitment  to  requirements  from  project  participants. 

•  SP  1 .3  Manage  changes  to  requirements  as  they  evolve  during  the  project. 

•  SP  1 .4  Maintain  bidirectional  traceability  among  requirements  and  work  products. 

•  SP  1 .5  Ensure  that  project  plans  and  work  products  remain  aligned  with  the  requirements. 

•  GP  2.1  Establish  and  maintain  an  organizational  policy  for  planning  and  performing  the 
process. 

•  GP  2.2  Establish  and  maintain  the  plan  for  performing  the  process. 

•  GP  2.3  Provide  adequate  resources  for  performing  the  process,  developing  the  work  products, 
and  providing  the  services  of  the  process. 

•  GP  2.4  Assign  responsibility  and  authority  for  performing  the  process,  developing  the  work 
products,  and  providing  the  services  of  the  process. 

•  GP  2.5  Train  the  people  performing  or  supporting  the  process  as  needed. 

•  GP  2.6  Place  selected  work  products  of  the  process  under  appropriate  levels  of  control. 

•  GP  2.7  Identify  and  involve  the  relevant  stakeholders  of  the  process  as  planned. 

•  GP  2.8  Monitor  and  control  the  process  against  the  plan  for  performing  the  process  and  take 
appropriate  corrective  action. 

•  GP  2.9  Objectively  evaluate  adherence  of  the  process  and  selected  work  products  against  the 
process  description,  standards,  and  procedures,  and  address  noncompliance. 

•  GP  2.10  Review  the  activities,  status,  and  results  of  the  process  with  higher  level  management 
and  resolve  issues. 
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leading  fhe  Convergence  of  Nofronol  Security  and  Technology1* 


Chutes  and  Ladders 

ISO/CMMI  Considerations 

November  16,  201 1 


Intro 


Since  our  company  was  ISO  9001 
certified,  we  assumed  we  were  all  set 
for  CMMI.  However,  much  like  the 
game  of  Chutes  and  Ladders,  we 
learned  when  we  performed  our  gap 
analysis  that  some  of  our  ISO 
practices  put  us  ahead  for  CMMI,  while 
others  meant  we  were  behind  where 
we  needed  to  be! 
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Topics  of  Discussion 


ManTech 

International  Corporation  * 


•  Challenges  faced 

-  Become  ISO  9001 :2008  certified 

-  Simultaneously  implement  CMMI  Maturity  Level  3 

•  The  gap  analysis  “reveal” 

-  “ladders”  (areas  where  ISO  procedures  and  practices  gave  us 
the  inside  edge) 

-  “chutes”  (areas  which  required  improvement) 

•  The  cultural  shift  required 

-  How  a  group  of  creative,  talented  and  opinionated  employees 
worked  together  to  adopt  processes,  procedures,  and  best 
practices 

•  The  road  less  traveled 

-  How  ISO  and  CMMI  learned  to  successfully  co-habitat  (or  at 
least  not  kill  each  other) 

-  Let’s  save  some  time  (or  helpful  hints) 


Background 


ManTech 
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•  The  Customer 
-Defense  Commissary  Agency 

•  The  SSESS  Team 
-34  employees 

•  The  System 

-DeCA  Interactive  Business  System 
(DIBS) 


Background  (continued) 


The  Contract 

-  ISO  9001  OR  CMMI 
DEV 

The  Challenges 

-  Documentation 

-  Procedures 

THE  SOLUTION 

-  ISO  9001 


ManTech 

International  Corporation  * 


The  Implementation... 
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•  In  2007,  we  started  work  on  the 
SSESS  contract. 


•  With  scarcely  documented  procedures, 
we  rolled  up  our  sleeves  and  got  to 
work! 

•  We  had  34  people  who  were: 

-  Overworked 

-  Overwhelmed 

-  Working  like  crazy 


•  How  did  we  implement  ISO  9001 
during  all  this  chaos? 


First  Things  First 

Top  down  or  bottom  up? 

-  Without  management’s 
support,  no  one  would 
listen 

-  Without  employee  buy- 
in,  no  one  would  act 

Lesson  learned: 

-  Stakeholder 
engagement!!! 


ManTech 
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Stakeholder  Engagement 


•  It’s  all  about  the  message 

-  Say  it  loud 

-  Say  it  often 

-  Say  it  again 

-  Repeat  steps  as  needed 
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Get  Ahead  of  the  Curve... and  Stay  There  International  Corporation  • 


•  Our  guidance 

-  Prior  to  the  first  external  audit,  our 
appraiser  recommended  a  shift 
from  ISO  9001 :2000  to 
9001:2008. 

•  Lesson  learned: 

-  Whenever  possible,  become 
certified  or  appraised  in  the 
upcoming  version  of  the  standard. 
You  will  benefit  by  getting  ahead 
of  the  curve. 


Enter  the  CMMI... 
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We’re  prepared... or  are  we? 

-  As  we  entered  the  gap 
analysis:  “We  already  have 
processes  and  procedures  in 
place!  This  should  be  a  piece 
of  cake!” 

-  During  the  SCAMPI  C,  our 
eyes  were  opened:  “This  is  no 
piece  of  cake! 


A  Not  So  Simple  Union 
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"England  and  America  are  two  countries 
separated  by  a  common  language."  George 
Bernard  Shaw 


•  Matching  the  ISO  standard  with  CMMI  process 
areas  is  conceptually  impossible 

•  Limited  mapping  between  ISO  9001  and  CMMI  ML  3 
with  some  similar  terminology: 

-  Configuration  Management  =  Configuration 
Management  (well,  sort  of) 

-  Quality  Assurance  =  Verification 

-  Requirements  =  Requirements  (RD/REQM) 


It’s  A  Model,  Not  a  Process 
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•  If  I  comply  with  ISO  9001 :2008  7.3.1 ,  do  I  conform 
with  the  TS  SG  2? 


ISO  9001L2008 

CMMI-Dev  1.3 

7.3.1-  Design  and 

Development  Planning 

Technical  Solution  SG2- 
Develop  the  Design 

•  What’s  an  interface  anyway? 


ISO  9001L2008 

CMMI-Dev  1.3 

7.3.1-  Manage  Interfaces 
(Design  and  Development 
Planning) 

Product  Integration  SP  2.2- 
Manage  Interfaces 

Ladder... 
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•  Processes  were  well  documented  and 
institutionalized  on  the  project 


Chute... 
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Processes  were  institutionalized  on  the  PROJECT! 
The  Organization  wants  me  to  do  what? 


-  Change. ..again 

-  Document... again 

-  Train... again 


V  P 


■ 


Hint:  the  miracle  of  Tailoring 


Ladder... 
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The  SSESS  customer  required  and  supported 
quality  standards  for  the  project  which  means  we 
already  had  their  buy-in 
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Chute... 


•  CMMI  does  require  the 
time  and  commitment  of 
the  project  team 

-  Project  leadership  had 
to  ensure  this  didn’t 
take  away  from 
deliverables  and 
scheduled  tasks 

•  Lesson  Learned:  If  it 
doesn’t  get  planned,  it 
doesn’t  get  done. 
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Ladder... 
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•  Since  SSESS  already  had  process  evangelists 
embedded  on  the  team,  it  was  easier  to  make  CMMI 
disciples 


Lesson  Learned:  Form  partnerships  between  the 
Organization’s  Quality  Group  and  project  Quality 
team  members  early.  It  takes  a  village  to  raise  a 
process  framework. 


Chute... 
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•  Cultural  shift 

-  “We  already  made  it  up  the  ISO  mountain!  Who 
needs  CMMI?” 


•  Lost  in  translation 


-  The  SSESS  quality  team  was  often  serving  as  the 
translators  between  the  technical  staff  and  the 
Organization’s  corporate  quality  team  during  the 
CMMI  gap  analysis 


-  The  distillation  between  CMMI  and  ISO  often  left  our 
heads  spinning 

The  challenge  of  objectivity 


Shoot... let’s  just  do  it! 
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•  What  happens  when  you  marry  a  standard  and  a 
model? 

-  Continuous  quality  improvement,  of  course 
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Think  Globally.  Act  Locally. 
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•  We’re  in  Prince  George,  they’re  in  Northern 
Virginia... 

•  You  want  CMMI?  That’s  nice... what’s  in  it  for  me? 


What  Didn’t  Work... 
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•  Trying  to  match  up  ISO  and  CMMI  is  like  trying  to 
herd  cats... 


What  Didn’t  Work... 
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“Resistance  is  futile." 


Resisting  conformance  to  organizational 
standards... 
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What  Didn’t  Work... 


ManTech 

Internationa]  Corporation  * 


•  Not  planning  for  CMMI  tasks,  but  expecting  them  to 
get  done  anyway... 
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What  Didn’t  Work... 
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•  Jumping  into  CMMI  with  minimal  training... 


What  worked... 


•  Don’t  lose  buy-in:  from  the 
bottom  up  and  the  top  down 

-  Required  weekly  CMMI 
status  reports  to  many  levels 
of  ManTech  management 
with  regular  feedback 

-  CMMI  Day 

-  Team  engagement  with 
Mission  Assurance  (not  just 
the  PM/DPM) 

-  CMMI  Training 
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What  worked... 


•  Continual  reality  checks 

-  Why  are  we  doing  this? 

-  Is  it  working? 

-  What  can  we  do  better? 
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Questions  or  Comments 
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Note 


STRENGTH  T1IKOIGII INDLSTRV  &  TECHNOLOGY 


The  CMMI  Surveillance  Appraisal  method  described  in  this 
presentation  represents  the  thoughts  and  recommendations  of 
the  NDIACMMI  Working  Group,  with  inputs  from  selected 
stakeholders.  The  method  has  not  been  approved  or  endorsed 
by  the  CMMI  Program. 

You  feedback  will  help  determine  if  this  is  a  viable  concept. 
Suggestions  are  welcome. 


o 


| 


CMMI  Working  Group 


2 
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Introduction 
General  Concepts 
Operational  Concept 
Methodology 

•  Organizational  Unit  Change  Constraints 

•  Organizational  Scope  and  Sampling 

Case  Study  Example 
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Open  Issues 
Summary 
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Introduction 


STRENGTH  T1IKOIGII INDLSTRV  &  TECHNOLOGY 


SCAMPI  A  appraisal  costs  have  been  problematic  for  many 

businesses  seeking  to  use  CMMI  and  SCAMPI  A  to  benchmark 
their  organizational  processes. 

•  Appraisals  are  expensive  and  often  perceived  as  a  burden  to  CMMI 
adoption. 

After  a  SCAMPI  A  is  completed,  an  organization  must  then  conduct 
additional  SCAMPI  A  appraisals  every  3  years  to  maintain  the 
previously  achieved  rating. 

•  Diverts  valuable  process  improvement  funding  to  a  benchmarking  activity 
that  may  offer  little  value  in  return. 

Goal:  Develop  a  CMMI  Surveillance  Appraisal 

•  Remove  the  need  for  a  full  SCAMPI  A  appraisal  every  3  years  by  defining 
a  low  cost  surveillance  appraisal  method  to  extend  the  lifespan  of  a 
SCAMPI  A  rating  without  compromising  rating  integrity. 

•  Align  with  ISO  9001/AS9100,  TL9000  surveillance  benchmarking  activities. 
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STRENGTH  T1IKOIGII INDISTRV  &  TECHNOLOGY 


A  CMMI  surveillance  appraisal  should  be  not  be  more  than  25-30%  of 
the  cost  of  a  SCAMPI  A. 

•  This  should  be  a  design  constraint  on  the  method. 

A  CMMI  surveillance  appraisal  must  be  rigorous  enough  to  ensure  full 
confidence  in  extending  the  results  of  a  SCAMPI  A  appraisal. 

A  CMMI  surveillance  appraisal  will  provide  an  extension  of  SCAMPI  A 
results  for  2  years  from  the  completion  date  of  the  surveillance 
appraisal. 

Maximum  time  between  SCAMPI  A  appraisals  is  7  years. 

•  CMMI  surveillance  appraisals  cannot  extend  SCAMPI  A  rating(s)  beyond  7  years. 

•  Two  CMMI  surveillance  appraisals  could  be  conducted  in  that  timeframe 

CMMI  surveillance  appraisals  can  be  used  for  all  CMMI  constellations 
and  the  People  CMM.  Both  staged  and  continuous  representations 
will  be  supported. 
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General  Concepts  (2  of 2) 


STRENGTH  T1IKOIGII  INDtSTRV  &  TECHNOLOGY 


Model  scope  cannot  be  greater  than  the  SCAMPI  A  baseline  model 
scope. 

•  Can’t  add  PAs  that  weren’t  in  baseline,  including  previously  not  applicable  PAs. 

•  Can’t  increase  maturity  levels  or  capability  levels  using  a  CMMI  surveillance  appraisal. 

Organizational  unit  change  must  be  within  CMMI  surveillance 
appraisal  method  constraints. 

If  a  CMMI  surveillance  appraisal  identifies  problems  (e.g.,  goal 
failures)  the  SCAMPI  A  rating  cannot  be  extended. 

•  The  SCAMPI  A  rating  would  not  be  immediately  revoked. 

•  Additional  CMMI  surveillance  appraisals  would  not  be  permitted  until  after  another  SCAMPI  A. 

If  a  SCAMPI  A  rating  (or  successful  surveillance  appraisal  extension) 
expires  before  a  surveillance  appraisal  is  successfully  conducted, 
the  rating  cannot  be  restored  via  a  surveillance  appraisal. 

Maximum  duration  of  surveillance  appraisal  is  45  days. 

CMMI  surveillance  appraisal  ratings  should  be  noted  as  such  in  PARS. 
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Example  Appraisal 


STRENGTH  TIIKOIGII INDUSTRY  &  TECHNOLOGY 


TO:  T+1 

SCAMPI-A  Year 


T+2 

Years 


T+3 

Years 


T+4 

Years 


T+5 

Years 


T+6 

Years 


T+7 

Years 


1  1 

1  1 

i  i  1 

i  i  1 

_ ^ 

CMMI  ML  rating  validity 

1 

7 

T+2: 

T+4: 

Use  2  Surveillance 

Conduct 

Conduct 

Appraisals  to  extend 

Surv. 

Surv. 

rating  3  years 

Appraisal 

Appraisal 

Extended  CMMI  ML  rating  validity  Extended  CMMI  ML  rating  validity 


CMMI  ML  rating  validity 


Use  2  Surveillance 
Appraisals  to  extend 
rating  4  years 


T+3: 


T+5: 


Conduct 

Surv. 

Appraisal 


Conduct 

Surv. 

Appraisal 


|  Extended  CMMI  ML  rating  validity  |  Extended  CMMI  ML  rating  validity 
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Surveillance  Appraisals 

-  Operational  Concept 


STRENGTH  THKOK.il  INDUSTRY  &  TECHNOLOGY 


Time,  coverage,  confidence 

T-0: 


Cor 


objeci 
(sp|> 
ce 


textual  Overview 
tives,  process  plans 
nsor,  EPG,  QA  leads) 
measures 


Stage  2:  Hig  i 
•Work  products 
high  leverage 
•Interviews;  (ba£ 
as  needed 


Stage  3 


•Sampli 


Stage  1 : 

•Business 
•Interviews; 

•Performan 
•Process  tieplolyment/maintenance 
•QA  audit  analysis 


-Yield  Artifacts 
(basic  units,  support); 
artifacts 

ic  unit/support  leads 


Model  Sampling 


Artifacts,  affirmations  (interviews) 


T7 

<  T+3  yrs: 


?del  SGs,  GPs 


SCAMPI-A 


Surveillance  Appraisal 
Planning 


G) 
,  C 

c  'E 

i  | 

0. 


G) 
,  C 

c  'E 

i  J 

0. 


OU 


•Basic  Units 
•Support  Functions 


OU 


•Basic  Units 
•Support  Functions 


OU 


•Basic  Units 
•Support  Functions 


•Appraisal  scope  (model,  basic  units,  sampling) 
•Ratings 

•Findings  (strengths,  weaknesses,  opportunities) 

•Surveillance  appraisal  plan  (preliminary) 
•Schedule,  timelines,  constraints,  risks 
•SEI  notification  30  days  in  advance 
•Appraisal  scope,  initial  data  collection 


•Performance  measures 
•Process  deployment/maintenance 
•QA  audit  analysis 

•Scope:  initial  areas  of  emphasis  (PAs,  units,  etc.) 
•Data  collection  plan  for  Stage  2 


•Final  model  sample  scope:  investigative  and 
random  sampling 
(SGs,  GPs) 

•Final  data  coverage 

•Data  collection  plan  for  Stage  3 


•Practice  characterizations 
•Goal  ratings 

•Surveillance  appraisal  final  findings 
•Extension  of  Maturity  Level  rating  (+2  yrs  x  2) 
•Package  and  archive  appraisal  assets 
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Operational  Concept 

Stage  1 :  Contextual  Overview 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Stage  1 :  Contextual  Overview 
•Business  objectives,  process  plans 
•Interviews  (sponsor,  EPG,  QA  leads) 
•Performance  measures 
•Process  deployment/maintenance 
•QA  audit  analysis 


U) 

,  c 
'E  'c 
S  S 


ou 


•Performance  measures 
•Process  deployment/maintenance 
•QA  audit  analysis 

•Scope:  initial  areas  of  emphasis  (PAs,  units,  etc.) 
•Data  collection  plan  for  Stage  2 


Goals: 

Contextual  understanding  of  the  organization’s  process  state, 
process  deployment  and  maintenance  status,  possible 
noncompliance  trends 

Interviews: 

Sponsor,  EPG  leads,  Quality  leads 

Activities: 

Review  performance  measures  from  the  organization 

•  Business  objectives,  process  performance  objectives 

•  Measurement  and  analysis  related  to  achieving  objectives 

•  Corrective  actions  and  process  improvements 

Review  status  of  process  deployment  and  maintenance 

•  Status  of  deployment  to  basic  units,  support  functions 

•  Results  of  process  implementation/compliance  monitoring 

•  Status  of  improvement  activities,  appraisal  findings 

Review  PPQA  audit-type  analysis 

•  Trend  reports  of  audit  non-compliances 

•  Relate  significant  areas  of  concern  to  CMMI  process  areas 

Output: 

Potential  areas  for  further  investigation  in  Stages  2  and  3 

Model  Coverage 

(Conceptual) 


Typical  PAs: 

•MA,  PPQA 

•OPF,  OPD,  OT,  OPM,  OPP 
•Probe  selected  PAs  based  on 
data  collected 
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Operational  Concept 

Stage  2:  High  Yield  Artifacts 


STRENGTH  T1IKOIGII  INDUSTRY  &  TECHNOLOGY 


Stage  2:  High-Yield  Artifacts 
•Work  products  (basic  units,  support); 
high  leverage  artifacts 
•Interviews  (basic  unit/support  leads 
as  needed) 


U) 

,  c 

c  C 

OU 

s  s 

Q_ 

•Basic  Units 

•Support  Functions 

•Final  model  sample  scope:  investigative  and 
random  sampling 
(SGs,  GPs) 

•Final  data  coverage 

•Data  collection  plan  for  Stage  3 


Goals: 

Determine  Stage  3  model  sample  scope 

Interviews: 

Optional  interviews  with  basic  unit  and  support  function  leads 
as  needed  for  high  yield  artifact  analysis 

Activities: 

Review  example  high  yield  artifacts 

•  Plans,  documents,  reviews,  financial  reports,  etc.  (mdd  Table i 8) 

•  Collected  from  basic  units  and  support  functions  in  org  scope 
•Identify  areas  of  concern,  initial  Stage  3  model  sample  scope 

Use  random  sampling  to  fill  Stage  3  model  sample  scope 

•  Random  sampling  of  SGs/GPs  to  supplement  evidence  review 

•  Must  meet  method-defined  minimum  model  sampling  reqts 

•  Status  of  improvement  activities,  addressing  appraisal  findings 

“Finalize”  Stage  3  model  sample  scope 

•  Appraisal  team  can  identify  additional  SGs/GPs 

•  Determine  data  coverage  reqts  based  on  model  scope  and 
applicable  MDD  coverage  rules 

•  Update  Data  Collection  Plan  for  Stage  3 

Output: 

Final  Stage  3  model  scope,  updated  data  collection  plan 

Model  Coverage 

(Conceptual) 


•Map  OE  to  model  SPs,  GPs 
•Identify  model  sample  scope  for 
Stage  3  (see  sampling  rules) 


[  -Targeted  sampling  (investigative)  | 
[  -Random  sampling  (supplementary)! 
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Operational  Concept 

Stage  3:  Model  Sampling 


STRENGTH  T1IKOIG1I  INDUSTRY  &  TECHNOLOGY 


Stage  3:  Model  Sampling 

o> 

ou 

•Artifacts,  affirmations  (interviews) 

.  .E 
'E  c 

•Sampling  of  model  SGs,  GPs 

n~ 

•Basic  Units 

•Support  Functions 

•Practice  characterizations 
•Goal  ratings 

•Surveillance  appraisal  final  findings 
•Extension  of  Maturity  Level  rating  (+2  yrs  x  2) 
•Package  and  archive  appraisal  assets 


Goals: 

Rate  SGs  and  GGs  included  in  model  sample  scope  to  determine  if 
SCAMPI  A  rating  can  be  extended 

Interviews: 

As  needed  for  affirmations  based  on  applicable  MDD  coverage  rules 

Activities: 

Verify  objective  evidence 

•  Review  data  collected  by  OU  based  on  model  sample  scope 
and  MDD  coverage  rules 

•  Conduct  interviews  for  affirmations 

Perform  practice  characterizations  as  described  in  MDD 

•  Characterize  all  SPs  for  SGs  in  model  sample  scope 

•  Characterize  GPs  in  model  sample  scope 

Derive  findings  and  rate  goals 

•  Document  preliminary  findings  (strengths  and  weaknesses) 

•  Validate  preliminary  findings  (optional) 

•  Rate  goals  in  model  sample  scope,  determine  ratings  extension 

Report  results 

•Deliver  final  findings.  Package  and  archive  appraisal  assets. 

Output: 

Final  findings  and  ratings 

Model  Coverage 

(Conceptual) 


•Collect  and  verify  OE  for  SPs  and 
GPs  in  model  sample  scope 
•See  details  for  sampling  and  rating 
methods 

•Follow  MDD  rules  for  practice 
characterization  and  aggregation 
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Organizational  Unit  Change 
Constraints 


STRENGTH  T1IKOIGII INDISTRV  &  TECHNOLOGY 


If  the  Organizational  Unit  has  evolved  since  the  previous  SCAMPI  A... 

•  The  determinant  on  whether  a  surveillance  appraisal  is  appropriate  is  the  amount  of 
change  in  the  sampling  factors  used  to  scope  the  previous  SCAMPI  A. 


Change  in  Sampling  Factors/Values 

Surveillance  Appraisal  Candidate* 

No  change. 

Yes 

#  sampling  factors/values  has  been 
reduced,  with  no  new  sampling 
factors/values  added 

Yes 

#  sampling  factors/values  have  grown  or 
sampling  factors/values  have  changed 

No 

Internal  reorganizations  or  external 

Maybe 

mergers/acquisitions 

•Sampling  factor/value  changes?  -  see  above 
•If  no  process  implementation  impact  -  Yes 

*  -  The  lead  appraiser  makes  the  final  decision  on  whether  a  CMMI  surveillance  appraisal 
may  be  performed.  Document  rationale  in  plan/ADS. 
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Organizational  Scope  and 
Sampling  (iot2) 

Basic  Unit  and  Support  Function  Sampling 


STRENGTH  T1IKOIGII  INDUSTRY  &  TECHNOLOGY 


•  Organizational  scope  is  determined  using  SCAMPI  A  VI  .3  MDD  rules. 

•  Sampling  factors  beget  subgroups  which  beget  sample  basic  units  and 
support  functions. 

•  Sampled  basic  units  and  support  functions  may  or  may  not  be  the  same  as 
those  sampled  in  baseline  SCAMPI  A. 

•  All  sampled  basic  units  in  each  subgroup  and  support  functions  undergo 
Stage  2  high  yield  artifact  review  and  Stage  3  sampling. 

•  Stage  3  data  coverage  is  based  upon  applicable  SCAMPI  A  VI  .3  MDD  rules 
applied  to  Stage  3  model  sample  scope. 

•  Coverage  Rule  1  for  Process  Areas  is  not  applicable  by  design  of  the  CMMI 
surveillance  appraisal  method. 

•  Process  Area  practice  coverage  is  determined  by  SGs  and  GPs  identified  during 
Stages  1  and  2. 
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Organizational  Scope  and 
Sampling  (2 of 2) 


STRENGTH  T1IKOIGII 1M1LSTKV  &  TECHNOLOGY 


Model  Sampling  Methodology 

•  Specific  Goals  and  Generic  Practices  are  identified  in  Stage  1  and  2  for 
inclusion  in  Stage  3  sample  scope  (investigative  sampling). 

•  A  minimum  of  33%  of  SGs  and  GPs  in  model  scope  must  be  included  in  Stage 
3  sample  scope. 

•  Random  sampling  of  SGs  and  GPs  in  model  scope  supplements 
investigative  sampling  performed  in  Stagel  and  Stage  2. 

•  At  least  two  SGs  must  be  selected  for  random  sampling  (even  if  Stage  1 
and  Stage  2  identified  SGs  are  >  33%. 

•  At  least  one  SG  from  each  Process  Area  Category  (within  model  scope) 
must  be  included  in  the  Stage  3  sample. 

•  GPs  identified  for  Stage  3  sample  are  sampled  for  all  Process  Areas  with 
SGs  in  Stage  3  sample  scope. 

•  Appraisal  team  can  always  request  additional  evidence  beyond  original  sample 
scope. 
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Case  Study  Example  (iof5) 


STRENGTH  T1IK01GII INDLSTRV  &  TECHNOLOGY 


Reference  MDD  VI. 3  Appendix  F,  Case  Study  1  (expanded  to 

Maturity  Level  5).  OU  undergoes  a  CMMI  Surveillance  Appraisal 
2  years  after  achieving  Maturity  Level  5. 

•  Stage  1 

•  Team  meets  with  Sponsor 

•  Observes  business  objectives  unchanged  since  last  appraisal 

•  Team  meets  with  EPG  Lead,  Quality  Leads 

•  Observes  some  updated  OSSP  standards  and  quality  issues  with  suppliers 

•  Stage  2 

•  Team  reviews  OSSP,  project  plans,  tailorings,  status  reporting  packages 

•  Observes  QPPOs  unchanged  and  some  projects  not  meeting  objectives 
without  appearing  to  take  action  or  use  CAR 

•  Observes  supplier  quality,  cost,  and  schedule  problems  in  status.  Supplier 
stakeholder  involvement  questionable. 

•  Stage  3 

•  Team  identifies  all  SAM,  OPP,  QPM,  CAR,  OPM  SGs,  and  IPM  SG2  (1 1  SGs),  and 
GP  2.2  and  GP  2.7  for  Stage  3  sample 

•  For  33%  SG  coverage  (16),  randomly  select  5  more  SGs 

•  For  33%  GP  coverage  (4),  randomly  select  2  more  GPs 
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Case  Study  Example  (2of5) 


STRENGTH  T1IK01GII INDLSTRV  &  TECHNOLOGY 


SGs  and  GPs  identified  during  Stages  1  and  2: 

•All  SGs  in  SAM,  OPP,  QPM,  CAR,  OPM,  and  also  IPM  SG2 
•GPs:  GP  2.2,  GP  2.7 
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Case  Study  Example  (3  of  5) 


STRENGTH  T1IK01GH 1NDLSTRV  &  TECHNOLOGY 


Use  Random  Number  Generator  to  select  5  additional  SGs  and  2  additional  GPs. 


PA 

SG 

#  SPs 

1 

CM 

SG  1 

3 

2 

CM 

SG  2 

2 

3 

CM 

SG  3 

2 

4 

MA 

SG  1 

4 

5 

MA 

SG  2 

4 

6 

PPQA 

SG  1 

2 

7 

PPQA 

SG  2 

2 

8 

PMC 

SG  1 

7 

9 

PMC 

SG  2 

3 

10 

PP 

SG  1 

4 

11 

PP 

SG  2 

7 

12 

PP 

SG  3 

3 

13 

REQM 

SG  1 

5 

14 

DAR 

SG  1 

6 

15 

IPM 

SG  1 

7 

16 

OPD 

SG  1 

7 

17 

OPF 

SG  1 

3 

18 

OPF 

SG  2 

2 

19 

OPF 

SG  3 

4 

PA 

SG 

#  SPs 

20 

OT 

SG  1 

4 

21 

OT 

SG  2 

3 

22 

PI 

SG  1 

3 

23 

PI 

SG  2 

2 

24 

PI 

SG  3 

4 

25 

RD 

SG  1 

2 

26 

RD 

SG  2 

3 

27 

RD 

SG  3 

5 

28 

RSKM 

SG  1 

3 

29 

RSKM 

SG  2 

2 

30 

RSKM 

SG  3 

2 

31 

TS 

SG  1 

2 

32 

TS 

SG  2 

4 

33 

TS 

SG  3 

2 

34 

VAL 

SG  1 

3 

35 

VAL 

SG  2 

2 

36 

VER 

SG  1 

3 

37 

VER 

SG  2 

3 

38 

VER 

SG  3 

2 

GPs 

1 

GP  2.1 

2 

GP  2.3 

3 

GP  2.4 

4 

GP  2.5 

5 

GP  2.6 

6 

GP  2.8 

7 

GP  2.9 

8 

GP  2.10 

9 

GP  3.1 

10 

GP  3.2 

11  investigative  sample  SGs  +  5  random  sample  SGs  =  16  =  33%  SG  model  scope. 
2  investigative  sample  GPs  +  2  random  sample  GPs  =  4  =  33%  GP  model  scope. 
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Case  Study  Example  (4  of  5) 


STRENGTH  T1IK0LGII 1M1LSTKV  &  TECHNOLOGY 


SCAMPI  A  SPs 

627 

SCAMPI  A  GPs 

1008 

Total 

1635 

MDD  Appendix  F  Case  Study  1  Table  27  expanded 
to  ML5  SCAMPI  A  Practice  Instantiations 


REQM 

PP 

PMC 

MA 

CM 

PPQA 

SAM 

OPF 

OPD 

OT 

RD 

Subgroups 

Sample 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

LA,  Commercial,  Large, 

Short 

1 

5 

12 

14 

12 

10 

12 

8 

12 

7 

12 

6 

12 

10 

12 

LA,  Commercial,  Small, 

Short 

1 

5 

12 

14 

12 

10 

12 

8 

12 

7 

12 

4 

12 

6 

12 

10 

12 

LA,  DoD,  Large,  Long 

1 

5 

12 

8 

12 

7 

12 

10 

12 

LA,  DoD,  Small,  Short 

2 

5 

12 

8 

12 

0 

0 

10 

12 

5 

12 

14 

12 

10 

12 

0 

0 

0 

0 

0 

0 

Dayton,  DoD,  Large,  Long 

1 

5 

12 

8 

12 

7 

12 

10 

12 

Dayton,  DoD,  Small,  Long 

2 

5 

12 

8 

12 

0 

0 

4 

12 

10 

12 

0 

0 

8 

12 

0 

0 

0 

0 

Enterprise  Process  Group 

9 

12 

7 

12 

7 

12 

35 

84 

42 

36 

30 

36 

56 

84 

28 

48 

8 

24 

12 

24 

9 

12 

7 

12 

7 

12 

60 

72 

TS 

PI 

VER 

VAL 

IPM 

RSKM 

DAR 

OPP 

QPM 

CAR 

OPM 

Subgroups 

Sample 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

LA,  Commercial,  Large, 

Short 

1 

8 

12 

9 

12 

8 

12 

5 

12 

10 

12 

7 

12 

6 

12 

7 

12 

5 

12 

LA,  Commercial,  Small, 

Short 

1 

8 

12 

9 

12 

8 

12 

5 

12 

10 

12 

7 

12 

6 

12 

7 

12 

5 

12 

LA,  DoD,  Large,  Long 

1 

8 

12 

9 

12 

8 

12 

5 

12 

10 

12 

7 

12 

6 

12 

7 

12 

5 

12 

LA,  DoD,  Small,  Short 

2 

8 

12 

9 

12 

8 

12 

5 

12 

6 

12 

5 

12 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Dayton,  DoD,  Large,  Long 

1 

8 

12 

9 

12 

8 

12 

5 

12 

6 

12 

5 

12 

Dayton,  DoD,  Small,  Long 

2 

8 

12 

9 

12 

8 

12 

5 

12 

6 

12 

5 

12 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Enterprise  Process  Group 

5 

12 

10 

12 

48 

72 

54 

72 

48 

72 

30 

72 

30 

36 

21 

36 

36 

72 

5 

12 

21 

36 

30 

72 

10 

12 
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Case  Study  Example  (5  of  5) 


STRENGTH  THKOLGH  INDUSTRY  &  TECHNOLOGY 


MDD  Appendix  F  Case  Study  1  Table  27  expanded 
to  ML5  Surveillance  Appraisal  Practices 


Total  Stage  3  SPs  Sampled 

161 

Total  Stage  3  GPs  Sampled 

144 

Total  Stage  3  Sample 

305 

REQM 

PP 

PMC 

MA  SGI 

CM  SG2 

PPQA  | 

lOPF  SG2 

OPD  SGI 

OT 

RD 

Subgroups 

Sample 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

SP 

GP 

LA,  Commercial,  Large, 

Short 

1 

* 

* 

* 

* 

* 

* 

4 

4 

2 

4 

* 

* 

6 

4 

* 

* 

LA,  Commercial,  Small, 

Short 

1 

* 

* 

* 

* 

* 

* 

4 

4 

2 

4 

6 

4 

* 

* 

LA,  DoD,  Large,  Long 

1 

* 

* 

* 

* 

* 

* 

4 

4 

2 

4 

* 

* 

LA,  DoD,  Small,  Short 

2 

* 

* 

* 

* 

* 

* 

4 

4 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Dayton,  DoD,  Large,  Long 
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Characterizations  and  Ratings 


STRENGTH  T1IKOIGII  INDUSTRY  &  TECHNOLOGY 


Characterizations  are  done  on  the  Stage  3  sampled  practices  using 
SCAMPI  A  MDD  characterization  rules. 

Goals  that  have  Stage  3  sampled  practices  are  rated  based  on  the 
characterizations  of  the  sampled  practices  and  associated 
weaknesses  (if  any). 

If  all  sampled  goals  are  satisfied,  the  SCAMPI  A  rating  can  be 
extended. 

If  any  sampled  goals  are  not  satisfied,  the  SCAMPI  A  rating  cannot 
be  extended. 
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Team  Qualifications  <iof2) 


mm  mwm  mm 

STRENGTH  T1IK01GII 1NDLSTRV  &  TECHNOLOGY 


CMMI  Surveillance  Appraisal  Team  member  qualifications  are  more 
stringent  than  SCAMPI  A  requirements 

•  Experienced  team  members  able  to  reach  accurate  conclusions  with  less  evidence 


SCAMPI  A  Surveillance  Appraisal 


SCAMPI  A 

Surveillance  Appraisal 

Minimum  team  size  is  4,  including  the  lead 
appraiser 

Minimum  team  size  is  2,  including  the  lead 
appraiser 

•Members  of  the  appraised  organization  are 
allowed  to  be  team  members  but  the  team  must 
have  a  minimum  of  2  team  members  external  to 
the  OU  (including  the  lead  appraiser). 

Team  members  complete  Introductory  to 
model  course 

Team  members  must  have  previous  experience 
as  appraisal  team  members  on  at  least  two 
SCAMPI  A  appraisals 

Average  of  6  years  field  experience  wrt 
reference  model,  aggregate  of  10  yrs 
management  experience,  1  team  member 
with  6  yrs  mgt.  experience 

Each  team  member  must  have  minimum  of  6  yrs 
field  experience,  (including  experience 
performing  practices  from  the  process  areas  that 
the  team  member  is  reviewing) 

Aggregate  of  25  years  field  experience 

N/A 

CMMI  Working  Group 
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Team  Qualifications 


High  Maturity 


SCAMPI  A 


Certified  High  Maturity  Lead  Appraiser 
(HMLA) 

All  members  of  high  maturity  mini-team 
have  high  maturity  experience 

HMLA  or  ATM  with  statistical  analysis  & 
other  high  maturity  training  assigned  to  high 
maturity  mini-teams 


Team  as  a  whole  has  collective  experience 
implementing  high  mat  activities 


CMMI  Working  Group 


(2  of  2) 


STRENGTH  TIIKOK.II  INDUSTRY  &  TECHNOLOGY 


Surveillance  Appraisal 


Certified  High  Maturity  Lead  Appraiser 
(HMLA) 

Team  members  reviewing  high  maturity 
process  areas  must  have  been  on  a 
previous  SCAMPI  A  high  maturity  appraisal 

At  least  one  team  member  reviewing  high 
maturity  process  areas  must: 

•  have  been  on  a  previous  SCAMPI  A  high 
maturity  appraisal  as  part  of  a  high  maturity 
mini-team  or 

•be  a  certified  HMLA  who  has  been  on  a 
high  maturity  appraisal  as  lead  or  team 
member 

N/A 
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Open  Issues 


STRENGTH  T1IKOIGII  INDUSTRY  &  TECHNOLOGY' 


VI. 2  Model  and  Method  Sunset  issue 

•  Should  CMMI  VI  .2  model  appraisals  be  candidates  for  surveillance 
appraisals? 

•  Under  discussion 

•  If  no,  there  may  be  no  surveillance  appraisal  market  until  2013 

•  If  yes,  outdated  model  ratings  are  being  extended 

•  Should  surveillance  appraisals  be  done  with  MDD  VI. 2? 

•  Recommendation  -  NO 

•  The  CMMI  Surveillance  Appraisal  method  requires  use  of  the  SCAMPI  A 
VI  .3  MDD  methods. 

•  Appraisal  scoping,  characterizations,  ratings,  etc.  must  be  done  in  accordance 
with  MDD  VI  .3 

•  For  example,  an  OU  that  used  MDD  VI  .2  would  have  to  convert  critical  factors  to 
sampling  factors,  and  apply  the  MDD  VI. 3  subgroup/sampling  process,  data 
coverage  rules,  etc. 

•  The  lead  appraiser  must  ensure  that  the  OU  in  the  CMMI  Surveillance  Appraisal 
falls  within  the  OU  change  constraints  described  earlier  in  this  presentation,  and 
document  the  rationale  in  the  CMMI  Surveillance  Appraisal  plan. 
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Summary 


STRENGTH  T1IK01GII INDLSTRV  &  TK'HNOLOGV 


Goal:  Define  a  low  cost  CMMI  Surveillance  Appraisal  method  to 
extend  the  lifespan  of  a  SCAMPI  A  rating  without  compromising 
rating  integrity. 


Lower  costs... 

...without  compromising 
integrity 

Less  evidence  collection  and  review 
by  the  organizational  unit  and 
appraisal  team 

Organizational  Unit  has  already 
undergone  a  recent  baseline 

SCAMPI  A  appraisal 

Fewer,  but  more  experienced  team 
members 

Team  member  qualifications  are 
more  stringent  than  SCAMPI  A  VI  .3 
MDD  team  member  qualifications 

Fewer  SCAMPI  A  appraisals 
performed  just  to  maintain  ratings 

Combines  investigative  sampling 
and  random  sampling  of  specific 
goals  and  generic  practices 

CMMI  Working  Group 
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For  More  Information 


mm mwm mm 

vi  ui  vn  rii  munr  1:11  huh  cruv  jh  nr:\* 


NDIA  CMMI  Working  Group 

http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/Pages/CMMI  Working  Group.aspx 
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Introduction 

Definitions 

•  Risks  (IEEE  Std  1540-2004;  Standard  for  Software  Life  Cycle  Processes) 

-  Program  and  project  risks  are  the  likelihood  of  an  event,  hazard,  threat,  or 
situation  occurring  and  its  undesirable  consequences 

•  Risk  (Project  Management  Body  of  Knowledge  PMBOK) 

-  An  uncertain  even  or  condition  that,  if  it  occurs,  has  a  positive  or  negative  effect 
on  project's  objectives 

•  Issues  (QATAR  National  Project  Management) 

-  An  issue  is  something  currently  happening  that  is  having  a  negative  impact  on  the 
project  and  requires  resolution  for  the  project  to  proceed  successful 

•  Issues 

—  An  issue  can  be  associated  with  a  risk  if  the  risk  is  realized;  has  occurred 

•  Opportunity  (The  American  Heritage  Dictionary) 

-  A  favorable  or  advantageous  combination  of  circumstances 

-  A  chance  for  progress  or  advancement 

•  Opportunity  (PMBOK) 

-  A  condition  or  situation  favorable  to  the  project,  a  positive  set  of  circumstances,  a 
positive  set  of  events,  a  risk  that  will  have  a  positive  impact  on  project  objectives, 
or  a  possibility  for  positive  chances 
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Introduction 

Definitions 


•  Risk  Response 

-  The  process  of  developing  options  and  actions  to  enhance 
opportunities  and  reduce  threats  to  project  objectives  PMBOK 

-  Includes  Mitigation  and  Contingencies 

-  Includes  acceptance  of  the  risk  or  issue  consequence 

•  Mitigation 

-  Risk  mitigation  implies  an  elimination  or  reduction  in  the  probability 
of  risk  occurrence  PMBOK 


•  Contingency 

-  Issue  contingency  implies  an  elimination  or  reduction  of  the  impact  of 
issues  or  alternative  actions  taken 
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Introduction 


*  Managing  Risks ,  Methods  for  Software  Systems  Development;  Dr.  Elaine  M.  Hall,  SEI  Series  in  Software  Engineering 
** Focus  of  this  presentation 
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Where  Are  We 


•  Introduction 

•  Reasons  for  Risk  Management 

•  Risk  Management 

•  Questions/Discussion 

•  References 

•  Contact  Information 
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Reasons  for  Risk  Management 


•  When  developing,  delivering,  and  acquiring  systems  and 
products 

-  developers  and  acquirers  face  many  challenges 

•  Challenges  can  exist  with  many  items  and  activities: 

-  Cost 

-  Schedule 

-  Technical 

-  Management 

-  Programmatic 

-  Process 

-  Performance 

-  Others? 
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Reasons  for  Risk/Issue  Management 


•  Consequences  may  be  numerous  if  challenges  are  not  mitigated 

-  Cost  overruns 

-  Late  deliveries 

-  Technically  inadequate 

-  Programmatic  difficulties 

-  Irate  management 

-  Irate  customer 

-  Canceled  project 

-  Loss  of  market  share 

-  Missed  opportunities 

-  Others? 
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Reasons  for  Risk/Issue  Management 

•  There  are  solutions  for  an  organization  to  help  mitigate  these 
challenges 

-  Proper  program/project  management 

-  Proper  program/project  planning 

-  Program/project  monitoring  and  control 

-  Adequate  budgets 

-  Adequate  schedules 

-  Proper  requirements  development  and  management 

-  Contract  tracking  and  oversight 

-  Product  evaluation 

-  Performance  management 

-  Risk  management 

-  Quality  assurance 

-  Configuration  management 

—  Independent  Verification  and  Validation  (IV&V) 

-  Others? 


MITRE  Al Florence 


Compliance  with  CMMI 


Software  Engineering  Institute  (SEI)  Capability  Maturity 
Model  Integration  (CMMI)  _ 

-  CMMI  for  Development  vl. 3 

-  CMMI  for  Acquisition  vl.3 

-  CMMI  for  Service  vl.3 


All  have 

Risk  Management 


In  order  for  organizations  to  be  compliant  with  CMMI  they  need  to 

establish  risk  management  capabilities 
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Where  Are  We 


•  Introduction 

•  Reasons  for  Risk  Management 

•  Risk  Management 

•  Questions/  Discussion 

•  References 

•  Contact  Information 
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Risk  Management  Process 


•  Risk  Management  is  an  overarching  process  that  encompasses 

-  Risk  Planning 

-  Risk  Identification 

-  Risk  Analysis 

-  Risk  Response 

-  Risk  Monitoring  and  Control 

PMBOK 
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Risk  Management  Flow 
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Risk  Management  Planning 

•  Risk  management  planning  is  the  process  of  deciding  how  to 
approach  and  conduct  the  risk  management  activities  for  a  project 

•  Planning  is  important  to 

-  Ensure  the  level,  type  and  visibility  of  risk  management  are  commensurate 
with  both  the  risk  and  importance  of  the  project  to  the  organization 

-  Provide  sufficient  resources  and  time  for  risk  management  activities 

-  Establish  an  agreed-upon  basis  for  evaluating  risks 

•  Risk  planning  should  be  completed  early  during  project  planning 
PMBOK 
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Risk  Management  Team 

•  The  Risk  Managment  planning  activity  may  assign  a  Risk 
Management  Team  to  administer  the  Risk  Management  Program 

•  A  Risk  Manager  may  be  assigned  to  manage  the  Risk  Management 
Team 

•  A  Risk  Management  Board  may  be  chartered  to  review,  accept, 
decline,  transfer  and  escalate  risks 

•  Hierarchy  Governance  Boards  may  exist  for  escalation  of  risks  based 
on  thresholds 

•  Everyone  on  the  program/project  is  responsible  for  managing  risks 


The  level  of  this  implementation  depends  on  the  size,  scope, 
critically,  safety,  security,  etc.  of  the  application 
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Risk  Management  Plan 


•  Risk  management  planning  needs  to  be  part  of  project  planning 

•  A  risk  management  plan  can  be  a  stand  alone  plan  or  part  of  the  project 
plan 

•  The  risk  management  plan  needs  to  be  tailored  to  the  scope  of  the 
application 

•  The  concepts  provided  in  this  tutorial  can  be  used  to  develop  the  plan 


Risk  Management  Plan  Outline 


•  Introduction 

•  Project  Description 

•  Risks/Issue/Opportunity 


Risk  Monitor  and  Control 

Risk  Register 

Issue  Management 

Issue  Contingency 

Risk/Issue  Training 

Glossary 

References 


Descriptions 

•  Risk  Identification 

•  Risk  Analysis 

•  Risk  Response 


-  Risk  Acceptance 

-  Risk  Avoidance 

-  Risk  Transfer 

-  Risk  Escalation 


MITRE  Al  Florence _ -  Risk  Mitigation 
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Risk  Identification 


•  Risk  Identification  is  the  activity  that: 

-  Identifies  potential  and  current  risks 

-  Examine  elements  of  the  program  to  identify  associated  potential  root 
causes  of  risks 

-  Risk  identification  begins  as  early  as  possible  in  successful  programs 
and  continues  throughout  the  life  of  the  program 


•  Risk  can  be  associated  with  all  aspects  of  a  program;  e.g. 


Requirements 

Threat 

Security 

Technology  maturity 
Supplier  capability 


Design 

Schedule 

Cost 

Performance 

Etc. 
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Risk  Description 

•  A  well-written  risk  statement  contains  three  main 
components: 

—  Cause  -  The  negative  conditions  that  currently  exist  relative  to  the  risk 

•  Identification  of  root  cause(s)  of  the  risk 

•  This  provides  justification  that  a  risk  exists 

—  Probability  of  Occurrence  -  The  likelihood  of  the  occurrence  of  the  risk 

•  Within  a  future  time  frame 

•  Or  a  future  event 

-  Consequence  -  The  effect(s),  negative  impact(s)  to  the  program(s)  in 
case  the  risk  occurs 

•  The  consequence  should  be  related  to  at  least  cost,  schedule,  scope  and 
performance 

•  Consequence  could  also  result  in  opportunities  that  may  surface  in 
correcting  the  problems 
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Risk  Description 


•  The  risk  is  written  in  a  chain  of:  Cause:  IF;  THEN 

Example 

An  Interface  Working  Group  has  not  been  formed  and  a  plan  to  form  one 
does  not  exist. 

IF  key  stakeholders  cannot  agree  on  interface  protocol  by  11/14/2011; 
THEN  the  schedule  for  development  and  delivery  will  be  delayed  causing 
cost  overruns. 


NOTE:  The  cause  includes  assurance  that  the  reason  for  the  risk  is  valid.  I.e.,  is 
there  a  compelling  reasons(a  root  cause)  to  assume  that  stakeholders  cannot 
agree  on  the  interface  protocol  by  11/14/2011?  Not  just  pie  in  the  sky. 
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Risk  Description 


Risks  must  be  written  in  a  clear,  concise  and  unambiguous  fashion 

Words  and  phrases  that  may  have  confusing  and  multiple 
interpretations  must  be  avoided 


-  Adequate 

-  Limited 

-  Ad  hoc 

-  Near  real  time 

-  All 

-  Periodic 

—  Always 

-  Portable 

-  Appropriate 

-  Rapid 

-  Clearly 

-  Several  A,so 

—  Easy 

-  Slow  http 

-  Existing 

-  Small 

-  Fast 

—  Sometimes 

—  Flexible 

-  State  of  the  art 

-  Future 

—  Sufficient 

-  If  required 

-  Usable 

—  Immediately 

—  User-friendly 

-  Large 

-  Weight 

-  Light 

—  When  required 

http://www.ppi-int.eom/newsletter/SvEN-017.php#artide 
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Risk  Analysis 

•  The  risk  is  submitted  to  the  Risk  Management  Board 

•  The  risk  is  accepted  or  declined  by  the  Board 

—  If  declined  rational  is  conveyed  to  the  submitter 

•  If  accepted  the  Risk  Management  Board  assigns: 

-  A  Risk  Analyst  responsible  for  conducting  risk  analysis  on  assigned 
risks 

•  Supported  by  Subject  Matter  Experts  (SMEs) 

—  A  Risk  Owner  responsible  for  ensuring  risks  are  properly  managed 
throughout  their  life 

—  Risk  Analyst  and  Owner  could  be  one  in  the  same 
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Risk  Analysis  Components 

•  Risks  have  the  following  components: 

—  A  future  root  cause(s)  (yet  to  happen)  which 

•  if  eliminated  or  corrected,  would  prevent  a  potential  consequence  from 
occurring 

-  A  probability  of  occurrence  (or  likelihood) 

•  assessed  at  the  present  time  and  updated  when  necessary  of  the  future 
root  cause  occurring 

-  The  consequence  (or  effect/impact)  of  that  future  occurrence 

—  The  time  horizon  during  which  the  consequences  will  occur  if  the  risk  is 
not  mitigated 

—  Risk  Priorities 

•  Mapping  of  probability  of  risk  occurrence  and  risk  consequence 

-  Risk  Triggers 

•  Specific  events  or  conditions  that  indicate  when  to  develop  and  execute 
mitigation  or  contingency  strategies 
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Risk  Analysis 

*  Qualitative  Risk  Analysis 


-  Relative  measure  of  risk  or  asset  value  based  on  ranking  or  separation  into  descriptive 
categories  such  as  low,  medium,  high;  not  important,  important,  very  important;  or  on  a 
scale  from  1  to  10. 

BusinessDictionary.com 

-  An  examination  and  prioritization  of  the  risks  based  on  their  probability  of  occurring  and 
the  impact  on  the  project  if  they  do  occur.  Qualitative  risk  analysis  guides  the  risk 
reaction  process. 

pmpbank.googlepages.com/glossary 

*  Quantitative  Risk  Analysis 

-  Incorporates  numerical  estimates  of  frequency  or  probability  and  consequence.  In 
practice,  a  sophisticated  analysis  of  risk  requires  extensive  data  which  are  expensive  to 
acquire  or  often  unavailable.  Fortunately,  few  decisions  require  sophisticated 
quantification  of  both  frequency  and  consequences 

-  Shortly  spoken  one  might  say  that  "quantitative  risk  analysis  breaks  down  risks  from  a 
high  medium  low  ranking  to  actual  numerical  values  and  probabilities  of  occurrence"  for 
being  able  to  compute  the  overall  effects 

(comp.  CR0SSWIND7.  p.  423) 
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Root  Causes 


•  A  future  root  cause  is  the  most  basic  reason  for  the  presence  of  a 
risk 

•  The  cause  of  the  risk  has  to  be  isolated  and  defined 

—  Root  causes  should  be  initially  identified  when  risks  are  identified 

-  Once  initial  root  cause  are  identified  they  may  need  to  be  analyzed 
further  to  determine  the  actual  deep  rooted  causes  of  the  risks 

-  Root  causes  are  documented  and  they  support: 

•  Establishing  risk  mitigation  and  contingency  strategies 

•  Improvement  opportunities 

•  Root  causes  can  also  be  referred  as  risk  drivers 


Root  Cause  Analysis.  An  analytical  technique  used  to  determine  the  basic 
underlying  reason  that  causes  a  variance  or  a  defect  or  a  risk.  A  root  cause  may 
underlie  more  than  one  variance  or  defect  or  risk.  ( (PMBOK®  Guide)  --  Fourth 
Edition)  Syn:  root-cause  analysis 
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Root  Causes 


•  Typical  root  causes  may 

Threat 

Requirements 
Technical  Baseline 
Test  and  Evaluation 
Modeling  and  Simulation 
Technology 
Logistics 


associated  with: 

Management 
Schedules 
External  Factors 
Budget 

Earned  Value  Management 
Production 
Industrial  Capabilities 
Cost 

"  Others? 
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Root  Causes 

Background  Information 

•  Threat  -  The  sensitivity  of  the  program  to  uncertainty  in  the  threat 
description,  the  degree  to  which  the  program  would  have  to  change 
if  the  threat's  parameters  change 

•  Requirements  -  The  sensitivity  of  the  program  to  uncertainty  in  the 
system  requirements 

•  Technical  Baseline  -  The  approved  and  fixed  configuration  of  a 
technical  item  at  a  specific  time  in  its  lifecycle  that  serves  as  a 
reference  point  for  change  control 

•  Test  and  Evaluation  -  The  adequacy  and  capability  of  the  test  and 
evaluation  program  to  assess  attainment  of  significant  performance 
specifications  and  determine  whether  the  system  is  operationally 
effective,  operationally  suitable,  and  interoperable  with  the  system 
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Root  Causes 

Background  Information 


•  Modeling  and  Simulation  -  The  adequacy  and  capability  of  M&S  to 
support  all  life-cycle  phases  of  a  program  using  verified,  validated,  and 
accredited  models  and  simulations 

•  Technology  -  The  degree  to  which  the  technology  proposed  for  the 
program  has  demonstrated  sufficient  maturity  to  be  realistically 
capable  of  meeting  all  of  the  program's  objectives 

•  Logistics  -  The  ability  of  the  system  configuration  and  associated 
documentation  to  achieve  the  program's  logistics  objectives  based  on 
the  system  design,  maintenance  concept,  support  system  design,  and 
availability  of  support  data  and  resources 
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Root  Causes 

Background  Information 


•  Management  -  The  degree  to  which  program  plans  and  strategies  exist 
and  are  realistic  and  consistent.  The  program  support  team  should  be 
qualified  and  sufficiently  staffed  to  manage  the  program 

•  Schedule  -  The  sufficiency  of  the  time  allocated  for  performing  the 
defined  tasks 

•  External  Factors  -  The  availability  of  resources  external  to  the  program 
that  are  required  to  support  the  program  such  as  facilities,  resources, 
personnel,  government  furnished  equipment,  etc. 

•  Budget  -  The  sensitivity  of  the  program  to  budget  variations  and 
reductions  and  the  resultant  program  turbulence 

•  Earned  Value  Management  (EVM)  -  The  adequacy  of  the  EVM  process 
and  the  realism  of  the  integrated  baseline  for  managing  the  program 
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Root  Causes 

Background  Information 


•  Production  -  The  ability  of  the  system  configuration  to  achieve  the 
program's  production  objectives  based  on  the  system  design, 
manufacturing  processes  chosen,  and  availability  of  manufacturing 
resources 

•  Industrial  Capabilities  -  The  abilities,  experience,  resources,  and 
knowledge  of  the  contractors  to  design,  develop,  manufacture,  and 
support  the  system 

•  Cost  -  The  ability  of  the  system  to  achieve  the  program's  life-cycle  cost 
objectives.  This  includes  the  effects  of  budget  and  affordability 
decisions  and  the  effects  of  inherent  errors  in  the  cost 
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Probability  of  Occurrence 

•  Probability  of  occurrence  assessed,  at  the  present  time,  is  the 
probability  of  a  future  root  cause  occurring 

•  The  chance  of  a  risk  occurring  is  rated  on  a  scale  between  >0 
and  1 

•  When  the  probability  of  occurrence  =  1;  (100%) 

-  The  risk  has  occurred;  it  then  becomes  an  issue  and  is  managed  as  an 
issue 

•  For  most  risks,  estimating  the  precise  probability  of 
occurrence  may  be  difficult 

-  Analysis  by  SMEs  may  be  necessary,  and  often  using  Best  Engineering 
Judgment 
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Probability  Scores 

•  Probability  of  occurrence  may  begin  with  a  qualitative 

description  of  probability,  which  will  tie  to  a  numeric  range  of 
probability. 


Sample  Risk  Probability  Scores 


Probability  Description 

Probability  %  of 
Occurrence 

Very  High  (Extremely  likely) 

>81%  and  =100% 

High  (Probable) 

61% -80% 

Medium  (Possible) 

41% -60% 

Low  (Unlikely) 

21% -40% 

Very  Low  (Highly  improbable) 

>1%  -  <20% 
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Consequence  of  Risk  Occurrence 

(Impact) 

•  Risks  are  reviewed  for  the  effect  that  they  would  have  on  the 
project's  objectives  and  other  elements  of  the  program 

•  The  level  of  impact,  may  be  rated  from  very  low  (1)  to  very 
high  (5),  and  is  assessed  against  at  least  four  categories: 

—  Cost 
—  Schedule 
—  Scope 
—  Performance 
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Consequence  of  Risk  Occurrence 


Program/Project 

Objective 

Very  Low 

Minor 

Low 

Moderate 

Medium 

Serious 

High 

Critical 

Very  High 

Catastrophic 

Cost 

Insignificant 

increase 

Increase 

<  2%  of 

budget  baseline 

Increase 

2-5%  of 
budget  baseline 

Increase 

6-10%  of 
budget  baseline 

Increase 

>  10%  of 
budget  baseline 

Schedule 

Insignificant 

slippage 

Slippage  <  2%  of 
project  baseline 
schedule 

Slippage  2-5%  of 
project  baseline 
schedule 

Slippage  6-10% 
of  project 
baseline 

schedule 

Slippage  >  10% 
of  project 
baseline 

schedule 

—  OR  — 

Slippage  past  a 
milestone 
mandated  by 
Congress 

Scope 

Scope  decrease 
barely  noticeable 

Minor  areas  of 
scope  affected 

Major  areas  of 
scope  affected 

Scope  reduction 
unacceptable  to 
sponsor 

Project  outcome 
is  effectively 
useless 

Performance 

Performance 
degradation 
barely  noticeable 

Performance 
degradation 
noticeable,  but 
does  not  fail 
acceptance 
criteria 

Performance 

reduction 
requires  sponsor 
approval 

Performance 

reduction 
unacceptable  to 
sponsor 

Project  outcome 
is  effectively 
useless 
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Risk  Exposure 

•  Risk  exposure.  (ISO/IEC  16085:2006  Systems  and  software 
engineering--Life  cycle  processes--Risk  management 

(1)  the  potential  loss  presented  to  an  individual,  project,  or 
organization  by  a  risk 

(2)  a  function  of  the  likelihood  that  the  risk  will  occur  and  the 
magnitude  of  the  consequences  of  its  occurrence 

•  Risk  exposure  can  also  be  called  Risk  Priority 

-  The  priority  of  a  risk  helps  to  determine  the  amount  of  resources  and  time 
that  should  be  dedicated  to  managing  and  monitoring  the  risk 

-  Very  Low,  Low,  Medium,  High,  and  Very  High  priority  is  assessed  by  using 
probability  and  impact  scores 

-  The  potential  timing  of  a  risk  event  may  also  be  considered  when 
determining  risk  management  actions 
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Risk  Priorities 


Impact 
Very  High 
High 

Medium 

Low 

Very  Low 


Very  Low  Low  Medium  High  Very  High 
Probability  of  Occurrence 


Very  Low  Low 

Priority  Priority 
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Medium  High  Very  High 

Priority  Priority  Priority 


35 


Risk  Priority  vs.  Mitigation/Contingency 


Very  Low 

Low 

Medium 

High 

Very  High 

Priority 

Priority 

Priority 

Priority 

Priority 

1 

1 

1 

1 

1 

Risk 

Monitor 

Monitor 

Monitor 

Monitor 

Watch 

List 

Develop 

Develop 

Mitigation 

Contingency 

Strategy 

Strategy 

Very  Low  Priority  Risks  are  placed  in  a  Risk  Watch  List  which  are 

periodically  monitored. 

Other  Risks  are  monitored  more  aggressively. 
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Identifying  Triggers 

•  Triggers  are  specific  events  or  conditions  that  indicate  when 
to  execute  mitigation  or  contingency  strategies 

•  Unless  a  condition  is  immediate,  a  trigger  should  be  defined 

•  Examples  of  triggers  may  include: 

-  Cost  performance 

-  Schedule  performance 

-  Results  of  management  reviews 

-  Occurrence  of  the  risk 

•  as  a  trigger  for  execution  of  contingency  strategies 
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Risk  Response 


•  Risk  response  is  the  process  of  developing  options  and 
determining  actions  to  enhance  opportunities  and  reduce 
threats  to  the  project's  objectives 

•  Risk  response  must  be 

-  Appropriate  to  the  significance  of  the  risk 

-  Cost  effective  in  meeting  the  challenge 

-  Timely  and  realistic  within  the  project  contend 

-  Agreed  to  by  all  parties  involved 
PMBOK 
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Risk  Response 


•  Risk  Responses  has  at  least  five  components 

-  Acceptance 

-  Avoidance 

-  Transfer 

-  Escalate 

-  Mitigate  (contingencies  -for  issues) 

•  Acceptance  -  Accept  the  consequences  of  the  risk  occurring 

-  Other  responses  may  not  be  possible 

-  Cost  to  respond  may  be  greater  than  the  benefit 

-  May  not  be  possible  to  prevent  the  impact  if  the  risk  occurs 

-  Impact  may  be  negligible 

-  Risk  may  be  imminent  and  should  be  handled  as  an  issue 
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Risk  Avoidance/Transfer 

-  Avoidance 

•  Eliminate  the  sources  of  high  risk  and  replace  them  with  a  lower- 
risk  alternative 

•  Avoid  risks  with  good  management  and  engineering  practices 

-  Transfer  -  Shift  the  responsibility  of  managing  and  resolving  the 
risk  to  another  party 

•  May  be  better  able  to  manage  the  risk 

•  May  be  the  proper  owner  of  the  risk 

•  Transfer  could  be  from  one  party  to  another  within  the  same 
organization 

•  Transfer  could  be  to  a  completely  different  organization 
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Risk  Escalation 

•  Escalation  -  Risks  should  be  managed  at  the  lowest  practical 
level 

-  But  conditions  may  arise  where  a  risk  should  be  escalated  to  higher 
levels  of  management  or  beyond  the  program/project 

-  The  next  higher  organizational  (Governance)  entity  may  be  able  to 
better  to  handle  the  risk/issue 

-  Thresholds  may  exist  that  determine  escalation 

•  Cost  of  impact 

•  Schedule  effect  of  Impact 

•  Scope  of  impact 

•  Performance  effect  of  impact 

•  Time  critical 

•  Cost  critical 
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Risk  Mitigation 


•  Taking  early  action  to  reduce  the  probability  and/or  impact  of 
a  risk  occurring  is  often  more  effective  that  trying  to  repair 
the  damage  after  the  risk  has  occurred 

•  Adapting  less  complex  processes,  conducting  more  tests,  or 
choosing  a  more  stable  supplier  are  examples  of  mitigation 
actions 

PMBOK 
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Risk  Mitigation 


•  The  following  are  important  guidelines  for  effective  risk 
mitigation: 

-  Prepare  detailed  mitigation  strategies  for  all  medium,  high 
and  very  high  risks 

•  With  sufficient  detail  about  what  is  to  be  done,  when,  where, 
and  by  whom 

-  Develop  mitigation  strategies  as  early  as  possible,  allowing 
time  to  address  risks  needing  special  attention  or  action 

•  Helps  reduce  the  chance  of  having  high-priority  risks  appear 
at  the  last  moment  on  the  critical  path 

-  Prepare  contingency  strategies  for  all  high  and  very  high 
priority  risks  and  risks  imminent  to  occur 


MITRE  Al Florence 


Risk  Mitigation 

Background  Information 

•  Adaptations  of  the  following  strategies  can  be  applied  to  a 
range  of  risks.  This  list  is  intended  merely  as  a  starting  point 
for  thinking  about  risk  mitigation 

-  Multiple  Development  Efforts  -  Create  competing  systems  in  parallel 
that  meet  the  same  scope  and  performance  requirements 

-  Alternative  Design  -  Create  a  backup  design  option  that  uses  a  less 
risky  approach 

-  Trade  Studies  -  Conduct  studies  to  arrive  at  the  least  risky  solution 

-  Early  Prototyping  -  Build  and  test  prototypes  early  in  the  system 
development 

-  Incremental  Development  -  Design  with  the  intent  of  upgrading 
system  parts  in  the  future 


MITRE  Al Florence 


44 


Risk  Mitigation 

Background  Information 

-  Technology  Maturation  Efforts  -  Normally,  technology  maturation  is  used 
when  the  desired  technology  will  replace  an  existing  technology  that  is 
available  for  use  in  the  system 

-  Robust  Design  -  This  approach,  while  it  could  be  more  costly,  uses 
advanced  design  and  manufacturing  techniques  that  promote  quality 
through  design 

-  Reviews,  Walk-Throughs,  and  Inspections  -  These  three  actions  can  be  used 
to  reduce  the  probability/likelihood  and  potential  consequences/impacts 
of  risks  through  timely  assessment  of  actual  or  planned  events 

-  Design  of  Experiments  -  This  engineering  tool  identifies  critical  design 
factors  that  are  sensitive,  and  therefore  potentially  high-risk,  to  achieve  a 
particular  user  requirement 
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Risk  Mitigation 

Background  Information 

•  Open  Systems  -  Carefully  selected  commercial  specifications  and 
standards,  which  can  result  in  lower  risks 

•  Use  of  Standard  Items/Software  Reuse  -  Use  of  existing  and  proven 
hardware  and  software,  where  applicable,  can  substantially  reduce  risks 

•  Use  of  Mock-Ups  -  The  use  of  mock-ups,  especially  man-machine  interface 
mock-ups,  can  be  used  to  conduct  early  exploration  of  design  options 

•  Modeling/Simulation  -  Modeling  and  simulation  can  be  used  to  investigate 
various  design  options  and  system  requirement  levels 

•  Key  Parameter  Control  Boards  -  The  practice  of  establishing  a  control 
board  for  a  parameter  may  be  appropriate  when  a  particular  feature  (such 
as  system  weight)  is  crucial  to  achieving  the  overall  program  requirements 
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Risk  Monitoring  and  Control 


•  In  order  to  effectively  monitor  and  control  risks  a  Risk 
Repository  needs  to  be  established 

-  Also  called  a  Risk  Register 

•  There  are  many  risk  tools  that  provide  repository  capabilities: 

-  Home  developed  tools 

-  Commercial  tools 

-  Corporate/agency  tools 


Note:  Risk  register  implementation  may  depend  on  project  size.  A 
month  long  project  might  just  need  a  spread  sheet  table  whereas  a 
multi-year,  geographically  dispersed  project  may  require  an  internet  and 
SQL-based  database  tool. 


MITRE  Al Florence 


Where  Are  We 


Introduction 

Reasons  for  Risk  Management 
Risk  Management 
Questions/Discussion 
References 
Contact  Information 


MITRE  Al Florence 


MITRE  Al  Florence 


49 


References 


•  IEEE/EIA  12207.2-1997  Annex  L—Risk  Management  Implementing  a  Risk 
Management  Process  for  a  Large  Scale  Information  System  Upgrade  -A 
Case  Study;  Paul  R.  Garvey,  The  MITRE  Corporation,  INCOSE/PMI  Risk 
Management  Symposium  9  &  10  May  2001,  INCOSE  INSIGHT,  Vol  4. 

Issue  1,  April  2001 

•  Managing  Risks ,  Methods  for  Software  Systems  Development SE I  Series 
in  Software  Engineering,  Elaine  M.  Hall,  1998  Addison-Wesley 

•  Reducing  Risks  with  the  Proper  Specification  of  Requirements;  Al 
Florence;  Risky  Requirements,  Crosstalk,  The  Journal  of  Defense  software 
Engineering,  April  2000 

•  Project  Management  Body  of  Knowledge  (PMBOK ) 

•  Issue  Management  Plan  Preparation  Guidelines;  QATAR  National  Project 
Management 


MITRE  Al  Florence 


50 


References 


•  Capability  Maturity  Model  Integration  (CM Ml) 

—  CMMI for  Development  vl. 3 

—  CMMI  for  Acquisition  vl.3 
—  CMMI  for  Service  vl .  3 
Software  Engineering  Institute  (SEI) 

•  IEEE  Std  1540-2004 ,  IEEE  Standard  for  Software  Life  Cycle  Processes— 
Risk  Management;  IEEE 

•  Issue  Management  Plan  Preparation  Guidelines;  QATAR  National  Project 
Management 

•  The  Black  Swan:  The  Impact  of  the  Highly  Improbable;  Nazism  Nicholas 
Tale  ;  The  Random  House  Publishing  Company 

•  http://pascal.computer.org/sev  display/index. action  SEVOCAB: 
Software  and  Systems  Engineering  Vocabulary 


MITRE  Al Florence 


Contact  Information 


Al  Florence 

florence@mitre.  org 
703  395  8700  -  Cell 
303  955  2286  -  Home 


52 


CMMI®  Training  - 
Avoiding  Waste  and 
Ineffectiveness 


Sari  D.  Schneider 

sari.d.schneider@boeing.com 

Defense,  Space  &  Security 
Boeing  Military  Aircraft 
Mobility  -  Philadelphia 
November  17,  2011 


BOEING  is  a  trademark  of  Boeing  Management  Company. 
Copyright© 201 1  Boeing.  All  rights  reserved. 


CMMI®  Training  -  Opportunities  or  Pitfalls? 


Budget 

CMMI  Definitions 
Corporate  culture 
Evaluations 
Employee  skill  base 
Goals 

Management 
Nature  of  the  business 


Previous  experience 

Product  complexity 

Processes 

Roles 

Skills 

Training  Tools 
Training  administration 
etc 


BOEING  is  a  trademark  of  Boeing  Management  Company 
Copyright  ©2011  Boeing.  All  rights  reserved. 


2  of  36 


CMMI  Training  Kaleidoscope 


Budget 


Product  Complexity 


Corporate 

Culture 


Processes 


Employee 
Skill  Base 


oals 


Previous^ 

ixperience 


Training 

Admin 


Nature 
of  the 
Business 


Evaluations 


Training!  Management 
\  Tools  A 


Skills 


CMMI  Definitions 


CMMI  Training  Kaleidoscope 


Employee  m/  \ 
Skill  Base  Evaluations 


Corporate 

Culture 


Previous^ 

ixperience 


Management 


Skills 


Training 
\  Tools  i 


Nature  r 
>  of  the 
Business 


Training 

Admin 


Processes 


Budget 


CMMI  Definitions 


Product  Complexity 


CMMI  Training  Kaleidoscope 


CMMI  Training  Kaleidoscope 


Budget 


Product  Complexity 


Corporate 

Culture 


Processes 


Employee 
Skill  Base 


oals 


Previous^ 

ixperience 


Training 

Admin 


Nature 
of  the 
Business 


Evaluations 


Training  l  Management 
V  Tools  A 


Skills 


CMMI  Definitions 


CMMI  Training  Inter-relationship  Diagram 


Budget 


Employee 
Skill  Base 


Training 
Administration 


Evaluations 


Trainin 

Tools 


Management 


Product  Complexity 


orporate  Culture 


Processes 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Definitions 


CMMI  Training  Inter-relationship  Diagram 


Budget 


(mm 

>5&t&  ffiKg 


CMMI  Definitions 


Employee 
Skill  Bas 


Training 
Administration^^lfsi 


Processes 


Evaluations 


Trainin 

Tools 


Product  Complexity 


orporate  Culture 


Skills 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Training  Inter-relationship  Diagram 


Budget 


Employee 
Skill  Base 


Training 
Administration 


Evaluations 


Trainin 

Tools 


Management 


roduct  Complexity 


orporate  Culture 


Processes 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Definitions 


CMMI  Training  Inter-relationship  Diagram 


Budget 


Employee 
Skill  Base 


Training 
Administration 


Evaluations 


Trainin 

Tools 


Management 


roduct  Complexity 


orporate  Culture 


Processes 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Definitions 


CMMI  Training  Inter-relationship  Diagram 


Budget 


Employee 
Skill  Base 


Training 
Administration 


Evaluations 


Trainin 

Tools 


Management 


Product  Complexity 


orporate  Culture 


Processes 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Definitions 


CMMI  Training  Inter-relationship  Diagram 


Employee 
Skill  Bas 


Training 
Administration 


Evaluations 


Trainin 

Tools 


Budget 


Management 


Product  Complexity 

orporate  Culture 

Processes 


Previous 

Experience 


Goals 


Nature 
of  the 
Business 


CMMI  Definitions 
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CMMI®  Training  -  Navigating  the  Pitfalls 


■  Who  needs/provides  training? 

■  What’s  the  goal? 

■  Where/How  is  training  provided? 

■  When  is  training  needed? 

■  Why  is  training  needed/not  needed? 
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CMMI®  Training  -  Getting  it  Right 


Factors  in  Getting  It  Right 

■  Complexity  of  the  product 

■  Nature  of  the  business 

■  Expected  skill  base  of  the  employees 

■  Corporate  culture  &  organization 

■  Importance  of  CMMI  to  the  company 

■  Laws  &  regulations 
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CMMI®  Training  -  Typical  Training  Issues 


■  Administration 

■  Budget 

■  Business  Complexity 

■  Contracts 

■  Metrics 

■  Need  Identification  &  Communication 

■  Process/Skill/Tool  Training 

■  Process  Architecture  &  Definition 

■  Process  Format 

■  Physical  Plant 

■  Role  Definition 

■  Training  Tools 

"  Updates  &  Training 

■  Waivers 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Budget 

■  What  is  the  baseline  level  of  competency? 

■  What  is  the  expected  level  of  competency? 

■  Who  pays  for  what? 

-  process  or  skill  training 

-  process  or  tool  training 

-  process  updates  and  training  material  updates 


BOEING  is  a  trademark  of  Boeing  Management  Company 
Copyright  ©2011  Boeing.  All  rights  reserved. 


17  of  36 


Training  Administration  &  the  Other  Training  Issues 


Training  Administration  and  Business  Complexity 


Number  of  products 

Contract  requirements 

Legal  and  regulatory  requirements 

Multiple  levels  of  training  requirements 

Single  or  multiple  sites 

Size  of  the  workforce 

Diversity  of  the  workforce 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Contracts 

■  Budget  for  upgrade  training 

■  Whose  budget? 

■  Scheduling  of  training 

■  Cost  factored  into  the  proposal? 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Delivery  Methods 

■  On  the  Job  Training 
-Web 

-  Instructor-led 

■  Mixed  delivery 

■  In-house  or  external 

■  Simulations 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Metrics 

■  Format  -  individual,  project,  manager,  roll-up 

■  Whose  budget? 

■  What  do  the  metrics  tell  about  the  training? 

-  Quality 

-  Quantity 

-  Skill  level 


■  Who  needs  to  know  what,  why,  and  when? 

-  Course  completions 

-  Competency  levels 

-  Personally  Identifiable  Information 

■  What  are  the  success  criteria? 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  &  Need  Identification 

■  Identification  of  training  needs 

■  Communication  of  training  needs 

■  Need  to  maintain  currency 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  &  Processes 

■  Definition  of  ‘process’ 

-  Series  of  tasks  to  produce  an  output 

-  Process  or  skill? 

-Tool  instructions 

-  Multiple  sites 

■  Process  Hierarchy 

■  Process  Updates 

■  Process  Format 

■  Process  Inter-relationships 

■  Process  detail 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Skill/Tool  Training 

■  Definitions 

■  Budget 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Role  Definition 
■  Consistent  definition 


-  programs 

-  sites 

-  processes 

-  departments 

■  Level  of  detail  in  the  definition 

-  budget 

-  scheduling  of  training 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  the  Training  Tool 

■  Course  equivalents 

■  Course  waivers 

■  Course  updates 

■  Course  Curricula 

■  Number  of  courses 

■  Number  of  people 

■  Questionnaires 

■  Scheduling 

■  Cancellations 


BOEING  is  a  trademark  of  Boeing  Management  Company 
Copyright  ©2011  Boeing.  All  rights  reserved. 


26  of  36 


Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  the  Training  Tool  cont’d 
■  Number  of 

-  courses 

-  people 

-  sites 

-  roles 

-  levels  of  competency 

-  processes 
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-skills 

-  available  instructors 

-  available  training  venues 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Training  Tool 

■  Multiple  users 

■  Multiple  levels  of  access 

■  Multiple  levels  of  requirements 

-  Federal,  state,  and  local  laws  and  regulations 

-  Head  office,  division,  sites,  projects 

■  Budget 

■  Upgrades 

■  Training  on  how  to  use  the  tool 

■  COTS  or  DIY 
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Training  Administration  &  the  Other  Training  Issues 


■  Training  Administration  and  Updates 

■  Training  materials  are  part  of  the  tool/process/skill  level 
updates 

■  Budget  -  whose? 

■  Impact  of  out  of  date  training  materials 
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Summary  Training  Administration  and  CMMI® 


Training  Administration 

■  Is  inter-related  with  multiple  aspects  of  the  business 

■  Impacts  all  areas  of  training 
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Summary  Training  Administration  and  CMMI® 


■  Training  Administration 

■  Does  the  discipline  of  CMMI  assist  the  development  and 
maintenance  of  the  training  program?  In  other  words  the 
administration  of  the  training  program? 

-  How  is  that  impact  evaluated? 
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CMMI®  Training  -  Avoiding  Waste  and 

Ineffectiveness 


■  If  CMMI  were  no  longer  a  business  need,  would  you 
keep  doing  training  the  same  way? 

■  What  would  be  dropped? 

■  What  would  be  retained? 

■  What  are  the  criteria  for  making  these  decisions? 
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CMMI®  Training  -  Avoiding  Waste  and 

Ineffectiveness 


■  What  is  being  done  in  the  name  of  CMMI  that  is  not 
necessary? 

■  How  do  you  know  it’s  not  necessary? 

■  How  do  you  know  it  is  necessary? 
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CMMI®  Training  -  Avoiding  Waste  and 

Ineffectiveness 


■  Is  your  CMMI  implementation  being  done  in  the 
Leanest  way  possible? 

■  Is  it  being  driven  by  your  inefficiencies? 
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Integrated  Compliance  Approach  hp\RR!S 


CMMI  Model 
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Work  Products  for  Compliance 


Overview 

A  brief  description  of  the  process  intent 


Entry  Criteria 

Exit  Criteria 

State,  Prerequisites,  Criteria 

State,  Criteria 

Inputs 

Outputs 

Needed  work  products,  resources 

Resulting  work  products 

Required  Activities 

Mandatory  tasks  to  implement  the  process 


Measures 

Process  performance  against  plans 

Organizational  Improvement  Information 

Metrics,  reusable  work  products 

Verification 

Process  compliance  oversight 

Tailoring 

Approved  tailoring,  process  specific 

Implementation  Guidance 

Common  implementation  descriptions 

Supporting  Documentation  and  Assets 

Applicable  organizational  references 


Program  work  products 
needed  to  demonstrate 
IPM  process  compliance 
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Organizational  Process  Context 


Integrated  Process  Manual  (IPM) 


Practices 
Process  Areas 


Capability  Maturity  Model®  Integration  (CMMI®) 


QA 

Integrated  Process 
(IP) 

Audits 

for 

IPM 


Standard  CMMI® 
Appraisal  Method 
for 

Process 

Improvement 

(SCAMPISM) 
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Approach  and  Challenges 


•  Streamline  processes  based  upon  systemic  analysis 
of  compliance  and  user  feedback 

•  Balance  of  product-centric  vs.  process-centric 

•  Lean  Six  Sigma 

•  Continuous  streamlining  of  organizational  processes 
and  appraisals  in  parallel  while  maintaining: 

-  Compliance  to  organization  internal  requirements  and 
external  industry  standards  (e.g.,  CMMI®) 

-  Minimal  impact  to  organizational  performance 

•  Validation 

-  Internal  assessments 

-  SCAMPISM 
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Streamline  Process  Requirements  hp\RR!S 


•  Common  process  tailoring  across  programs 

•  IPM  statements  with  no  CMMI  relationship 

•  Consolidation,  modification  or  deletion  of  IPM  statements 
reducing  the  amount  of  work  products  without 
compromising  the  overall  process  requirements 

•  SCAMPI  results  to  identify  work  products  not  required 

Process  Requirements 
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Streamline  Process  Changes 


•  Entry/Exit  Criteria  and  Verification  sections  removed  from 
PCM  tool  in  each  process 

-  Programs  not  required  to  tailor,  provide  work  products,  or 
audit 

-  Remains  in  IPM  as  reference/guidance 

•  Deleted  procedural  detail 

•  Combined  similar/related  statements  within  a  process 

-  Examples 

•  Establishing  and  maintaining  plans,  budgets,  schedules,  etc. 

•  Identify  and  categorize  risks 

•  Consolidated  statements  within  a  process  that  had 
similar/same  expected  work  products  resulting  in  fewer: 

-  Statements  for  tailoring 

-  Work  products 

-  Audits  by  QA 
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Streamlining  Compliance 


Overview 


A  brief  description  of  the  process  intent 


Entry  Criteria 

State,  Prerequisites,  Criteria 

Inputs 

Needed  work  products,  resources 

Exit  Criteria 

State,  Criteria 


Outputs 

Resulting  work  products 


Required  Activities 

Mandatory  tasks  to  implement  the  process 


Measures 


Process  performance  against  plans 

Organizational  Improvement  Information 

Metrics,  reusable  work  products 

Verification 


Process  compliance  oversight 

Tailoring 

Approved  tailoring,  process  specific 

Implementation  Guidance 

Common  implementation  descriptions 

Supporting  Documentation  and  Assets 

Applicable  organizational  references 
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IPM  process  compliance 
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Streamline  Work  Products 


•  Product-centric  focus 

•  Reverse  engineering  to  achieve  simplification 

•  Reuse  of  unique  work  products 

•  Organization  default  work  products  and  locations 


Appraisal  Evidence 
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Reverse  Engineer  Artifacts  -  1 


•  Instead  of  looking  from  the  process  view  -  looked 
from  a  program  work  products  view 

•  Basic  guidelines: 

-  Every  CMMI®  practice  shall  have  a  minimum  set  of 
adequate  expected  work  products  in  PCM 

-  Every  I  PM  statement  shall  have  a  minimum  set  of 
adequate  expected  work  products  in  PCM 

-  Every  PCM  work  product  (existing  or  new)  shall  map 
to  one  or  more  IPM  statements  and  CMMI®  practices 

-  Maximize  the  re-use  of  existing  work  products 

•  PCM  Startup  Template 

•  Standard  Directory  Structure 
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Reverse  Engineer  Artifacts  -  2 


•  Mapped  program  work  products  to  I  PM  statements  and  to 
relevant  CM  Ml®  practices 

-  IPM  mapping  clearly  documented  in  PCM  tool 

-  CMMI®  mapping  in  PCM  tool  -  transparent  to  the  program 

•  Work  product  descriptions  clarified  to  help  the  program 
understand  relevance 

-  Descriptions  let  the  program  know  why  this  work  product  is 
important 

-  IPM  perspective 

-  CMMI®  perspective 

•  Provided  name  of  typical  project  work  product  to  be  used 

•  Provided  standard  directory  structure  location  where  that 
work  product  should  be  maintained 
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Process-Centric:  !PM  ^  WP^>  CMM I 


TlPM  Tag 

IPM  Statement 

Project  Work 
Product 

Expected  Work  Product  ^ 
Description 

jCMMITag  ^ 

RA.RA.3 

Work  with 
stakeholders  to 
capture  needs, 
expectations, 
constraints,  and 
interfaces  for  all 
phases  of  the 
system  life-cycle. 

Concept  of 
Operations 
(CONOPS) 

Vr 

Approved  Concept  of 
Operations  (CONOPS)  that 
documents  system  mission, 
system  operation,  operational 
control,  staffing,  interfaces 
and  operational 
environment  -  from  an 
external  perspective 

RD.GP2.6 

'rd.spi.i  j' 

Uustomer 

requirements 

specification 

Uustomer  requirements 
specification  (i.e.,  A-Spec) 

KLJ.UHZb 

RD.SP1.1 

RD.SP1.2 

Minutes  from  working 
groups 

Records  of  requirements 
elicitation  techniques  utilized 
on  the  program  (e  g., 
Prototypes,  modeling, 
simulation,  working  groups, 
use  cases) 

RD.SP1.1 
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Product-Centric:  WP~>  iPM  ->  CMMi  H  /?/?/S 


^Project  Work  Product 

IPM  Tag 

IPM  Statement  ^ 

CMMI  Tag 

Concept  of  Operations 
(CONOPS) 

RA.RA.3 

Work  with  stakeholders  to  capture 
needs,  expectations,  constraints,  and 
interfaces  for  all  phases  of  the  system 
life-cycle. 

RD.GP2.6 

RD.SP1.1 

RA.RA.4.a 

Ensure  the  stakeholder  mission  and 
operational  needs  are  validated  and 
documented  in  the  Concept  of 
Operations  (CONOPS). 

RD.SP3.1 

RD.SP3.4 

SAD.RA.3.g 

Define  the  system  functional  baseline. 

J 

RD.SP3.1 

^ - 

- ^ 

TS.SP1.2 
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Product-Centric:  WP->  CMMi  ->  iPM  hf/\RRIS 


Project  Work  Product 

CMMI  Tag 

CMMI  Practice 

|IPM  Tag 

Concept  of  Operations 
(CONOPS) 

RD.GP2.6 

Place  designated  work  products  of  the 
requirements  development  process  under 
appropriate  levels  of  contol. 

RA.RA.3 

RD.SP1.1 

Elicit  stakeholder  needs,  expectations, 
constraints,  and  interfaces  for  all  phases  of 
the  product  life  cycle.  ^ 

RA.RA.3 

KU.SP3.1 

hstablish  and  maintain  operational 

concepts  and  associated  scenarios. 

RA.RA.4.a 

3AD.RA.3.g 

RD.SP3.4 

Analyze  requirements  to  balance 
stakeholder  needs  and  constraints. 

RA.RA.4.a 

TS.SP1.2 

Select  the  product  component  solutions  j 
that  best  satisfy  the  criteria  established  .xl 

pAD.RA.3.g 
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Standard  Directory  Structure 


•  Supports  IPM  Compliance  with  work  products  in  a 
common  structure  across  programs 

•  Top  level  directories  are  used  as  location  for  program 
work  products 

-  Avoids  tying  PCM  work  products  to  low  level  directories 

-  Easy  access  by  all  program  team  members 

-  Avoids  confusion  as  to  which  is  the  latest  version  of  a  work 
product 

-  Flexibility  for  custom  directories  which  contain  “work-in-progress” 

•  Pre-populated  with  latest  forms,  checklists  and  plan 
templates 

-  Set  up  by  IT  group  when  program  data  server  is  assigned 
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Standard  Directory  Structure 
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Name 

Size  Type 

Date  Modified 

OCM_DM 

File  Folder 

1/19/2006  12:10  PM 

O  Contracts 

File  Folder 

4/3/2007  1:07  PM 

ODataJJbrary 

File  Folder 

1/31/2006  9:02  AM 

_  iElectrical_Engineering 

File  Folder 

1/19/2006  12:10  PM 

jIPT_[Name] 

File  Folder 

1/19/2006  12:10  PM 

‘^Manufacturing 

File  Folder 

1/19/2006  12:10  PM 

•Cj  Material_Management 

File  Folder 

1/31/2006  9:02  AM 

~l  Mechanical_Engineering 

File  Folder 

1/19/2006  12:10  PM 

Program_Controls 

File  Folder 

4/3/2007  1:04  PM 

iiii-i  Program_Management 

File  Folder 

1/19/2006  12:10  PM 

‘i-  ~i  Project_Engineering 

File  Folder 

2/24/2006  12:24  PM 

Jj  Quality_Assurance 

File  Folder 

1/19/2006  12:10  PM 

ii~_H  Sof  tware_Engineering 

File  Folder 

1/19/2006  12:10  PM 

^Subcontracts 

File  Folder 

1/31/2006  9:02  AM 

.  System  IandT 

File  Folder 

1/19/2006  12:10  PM 

Sy  stems_Engineering 

File  Folder 

2/15/2006  5:05  PM 

,u  Sy stems_5upport_Engineering 

File  Folder 

1/19/2006  12:10  PM 

Owner.txt 

1  KB  Text  Document 

4/29/2007  5:09  PM 

18  objects 


209  bytes  ©  Trusted  sites 
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Streamlining  Results 


if\nms 


•  Significant  reduction  in 
process  requirements 

•  Maximized  the  re-use  of 
appraisal  evidence  to 
minimize  the  number  of 
unique  work  products 

•  Created  a  process¬ 
centric  view  to  maintain 
program  work  products 

•  Reduced  effort  required 
by  programs 

•  Maintained  CMMI® 
compliance 
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Streamlining  Appraisal 


if\nms 


•  Problem 

-  Total  cost  of  SCAMPISM  for  division  is  significant  and 
increases  every  SCAMPISM  cycle  (3-years) 

•  Goals 

-  Reduce  SCAMPISM  preparation  effort  using  Lean 
method 

•  Measurement 

-  SCAMPISM  preparation  effort 

•  Benefits 

-  More  efficient  SCAMPISM  preparation  process  with 
earlier  feedback  for  corrective  actions 


Streamlining  Process  Deployment  and  Compliance 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  23 

14-17  Novem  ber  20 1 1 


Approach 


•  Objective 

-  Reduce  effort  in  conversion  for  work  products  from  internal 
organizational  requirements  to  CMMI®  Practices 

-  Establish  a  work  product  priority  to  focus  on  the  number  of 
CMMI®  practices  affected  by  each  work  product 

-  Reduce  the  rework  in  discovering  the  correct  work  product 

•  Implementation 

-  Automate  the  conversion  process 

-  Prioritized  work  product  review 

-  Utilize  process  experts  to  data  mine  for  work  products 

-  Complete  improvements  prior  to  next  SCAMPISM 

-  Establish  more  detailed  measurements  of  SCAMPISM 
activities  for  future  improvements 

•  Validation 

-  SCAMPISM 


Streamlining  Process  Deployment  and  Compliance 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  24 

14-17  Novem  ber  20 1 1 


SIPOC 


Define 


_ N.r _ I\, _ Ns.  _ _ Ns. 

Measure  Analyze  Improve  Control 


Supplier 

-  SCAMPI 
Projects 

-  Organizational 
(HR,  DPG) 


Input 

-  SCAMPI 
Projects 

-  Program  Work 
Products 

-  Organizational 
Work 
Products 


Export,  conversion  and 
mapping  of  work  products 
is  Non-Productive 


Process 

-  Characterize 
and  reduce  the 
frequency  to 
review  each 
work  product 

-  Work  products 
into  PCM 

-  Export  from 
PCM  into  Excel 

-  Compare  deltas 
from  last  PCM 
export 

-  CMMI® 
conversion 
mapping  macro 

-  Review  &  identify 
corrective  actions 

-  Map  corrective 
actions  back  to 
PCM 


Output 

Customer 

-  Corrective  Actions 

-  Program  Team 

-  CMMI®  Progress 

-  Management 

Report 

-  Independent 

Appraiser 

Rework  in  discovering  the 
correct  work  products  is 
Non-Productive 
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Brainstorming 


Define 

•  Facilitated  session  with  team  resulting  in  42  items 
o  Identified  3  possible  Lean  applications 

■  Reduce  effort  in  PCM  to  CMMI®  conversion  for  projects  work  products 

■  Establish  a  work  product  priority  to  focus  on  the  items  that  typically 
have  issues  and  minimize  the  amount  of  effort  appraising 

■  Reduce  the  frequency  of  discovering  correct  work  products 

0N0  detailed  measurement  breakdown  available  from  previous  SCAMPISM 
components  or  subparts 

■  Planning 
^Preparation 

S  PCM  to  CMMI®  conversion 
S  Discovery  of  work  products 
□  Review  work  products  for  corrective  action 
S  CMMI®  to  PCM  conversion 

■  Conduct 

■  Closeout 

Limited  Historical  Data  to  Demonstrate  Measureable  Improvement 


Measure 


Anajyze  Improve  Control^ 
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Improvement  Basis 


Define 

PCM  to  CMMI  Conversion 

•  Sampled  4  months  of  SCAMPISM  effort  for  2  individuals  involved  in 
conversion  and  applied  50%  to  represent  best  estimate  of  time  spent 
o  Averaged  115  hours/month 


Measure 


Analyze 


I  m  p  rove  Control^> 


Work  Product  Priority 

•  Analyzed  the  number  of  CMMI®  practices  affected  by  PCM  default  work 
products  to  prioritize 

Work  Products  vs.  CMMI  Practices 


t— irvjm^-LnuDr-'-oocno*— irsiroLouDr^ooOLOuDr^orMro^-i-ocnrocx)^— 10000 
t— It— It— It— i*— i*— i*— i*— irNirMrsirMrorororororo^-^-LOLOCn 


Sample  to  Establish  Measureable  Improvement 
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Improvement  Approach 


Define  >1  Measure  >1  Analyze 


I  m  p  rove  Control^> 


•  PCM  to  CMMI®  conversion  for  projects  work  products 

-  Reduced  to  a  one  time  event 

-  Alternative  communication  used  for  corrective  actions 

•  Establish  a  work  product  priority 

-  Focus  on  the  items  that  typically  have  issues  and  minimize  the 
amount  of  effort  appraising 

•  Find  the  correct  work  products  the  first  time 

-  Utilized  process  experts  to  data  mining  based  upon  standard 
organizational  tools  and  standard  program  directory  structure 

-  Eliminated  “bring  me  a  rock” 


Piloted  on  Next  SCAMPISM  Event 


Streamlining  Process  Deployment  and  Compliance 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  30 

14-17  Novem  ber  20 1 1 


Implement  Process  Controls 


_ K 

Control 

•  Let’s  Not  Do  This  Again 

-  One  time  event  for  PCM  to  CMMI®  conversion  of  projects  work  products 

-  Establish  a  work  product  priority 

-  Utilized  process  experts  to  data  mining 

•  Setup  work  codes  to  measure  SCAMPISM  activities  for  future 
improvements: 

-  Planning 

-  Preparation 

-  Program  Support 

-  Reviews  (Readiness  &  On-Site) 

-  Closeout 


Define  Measure  Analyze  >j  Improve 


Continuous  Process  Improvement 
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SCAMPI  Labor  Savings 


SCAMPI  Labor  Analysis 


2008  DPG  2011  DPG  -•-2008  Actuals  -*-2011  ETC  -B-2011  Actuals 


Labor  Savings  =  60% 

Program  Staff  Hours  =  difference  between  Actuals  and  DPG. 
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SCAMPI  Cost  Savings 


$ 


SCAMPI  Cost  Analysis 


Cost  Reduction  Reasons 

•  LSS  for  SCAMPI  preparation 

-  Reduced  PCM  to  CMMI  conversion 

-  Work  Product  prioritization 

•  DPG  vs.  Programs  discovery  of  WPs 

•  Balanced  Program  selection  with 
SCAMPI  compliant  representation 

•  Program  IPM  process  compliance 


Closeout 


2008  Actuals 


2011  ETC 


a)Q-t;>OC-Qm 
30>irOa>rga)<5 
<(0°ZQ^>U-S 


•2011  Actuals 


Cost  Savings  =  47% 
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Summary 


•  Streamlined  the  organizational  processes  resulting  in  reduced: 

-  Number  of  process  requirements 

-  Amount  of  appraisal  evidence 

-  Effort  required  by  programs 

•  Established  a  product-centric  view  to  complement  the  existing 
process-centric  view  providing: 

-  Efficient  and  focused  project  data  collection 

-  Improved  support  for  projects  and  the  organization 

-  Integrity  of  the  appraisal  method  and  achievement  of  sponsor 
objectives 

•  Maximized  the  re-use  of  unique  work  products 

•  Minimized  the  impact  of  changes  to  the  programs 

•  Simplified  the  preparation  and  conduct  of  appraisals 

•  Maintained  the  process  compliance  requirements: 

-  Relevant  and  adequate  evidence 

-  Organizational  processes 

-  CM  Ml®  processes 


Streamlining  Process  Deployment  and  Compliance 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  34 

14-17  Novem  ber  20 1 1 


Contact  Information 


Harris  Corporation 
P.0.  Box  37 


http://www.harris.com/ 

SEI  Partner 


Melbourne,  Florida  32902-0037 


Gary  Natwick 
gnatwick@harris.com 

•  SEI-Certified  Introduction  to  CMMI®  Instructor 

•  Harris  SEI  Partner  Business  &  Technical  Point  of  Contact 


®  CMMI  is  registered  with  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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Armament  Software  Engineering  Center  (Armament  SEC) 
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Provide  RDECOM,  Joint  Munitions  & 
Lethality  Life  Cycle  Management 
Command,  TACOM  Life  Cycle 
Management  Command,  resident 
PEO/PMs  and  other  customers  a 
Center  of  Excellence  for  Software 
Engineering  and  Software 
Acquisition  support  services  for 
Army  Weapon  Systems,  Trainers, 
and  Combat  Support  Systems 
throughout  the  entire  system  life 
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Background 


m 


Serve  as  a  Center  of  Excellence  for  Software  Engineering 
&  Software  Acquisition  Support  Services 


S  AMC  Chartered  Life  Cycle  Software  Engr  Center 

v'  Customer  Focus  -  Exceptional  Life  Cycle  Mission 
Execution;  Flexible,  Agile,  Innovative  Workforce 

S  Technology  Innovation  Leader  -  Winner  2004,  2006  & 
2007  DoD  Top  5  Program  of  the  Year 

S  State-of-the-Art  Facilities,  Equipment,  Tools 

^  CMMI  Level  5  -  2006  First  in  DoD  for  SE,  SW  &  SS 

s  CMMI  Level  5  -  2010  Sole  Gov  Organization  to  achieve; 
only  Gov  Org  to  successfully  re-appraise 

S  Resident,  Agent  of  the  Certification  Authority  (ACA) 

s  A  30+year  legacy  of  developing  &  sustaining  SW- 
intensive  systems  to  our  warfighters... 


Benet  Labs 
\  Watervliet,  NY 


ARDEC  (WSEC) 
Picatinny,  NJ 


* 


Firing  Tables  & 
Ballistics  Div. 
Aberdeen,  MD 


i 
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Over  275  Organic  Software  Engineers  ~  50%  with  Advanced  Degrees 
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Is  there  anything  beyond  CMMI? 


Growing  &  Sustaining  a  High  Maturity  Organization . . . 


•  Establish  an  Enterprise  Performance  Improvement  Framework 
(PIF)  that ... 

-  Harmonizes  operations  in  a  multi-model  environment 

-  Migrates  to  an  organizational  set  of  standard  processes  that 
leverages  the  successes  of  individual  organizational 
achievements 

-  Facilitates  sharing  of  best  practices ,  organizational  processes 
and  process  assets 

-  Reduces  investment  redundancies 


Results-based  and  linked  to  improved  program  performance  ! 
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•  Currently  pursuing: 

-  Baldrige 

-  CMMI-DEV 

-  C MM I  -  SVC 

-  ISO  9001 

-  Lean  6a 

-  DO  -  178B 

•  Under  consideration: 

-  CM  Ml  -  ACQ 

•  Audit  &  Appraisals:  % 

-  ISO  9001  Audit  Compliance 

-  SCAMPI  CMMI  -  DEV 
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Recipient 
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Performance  Improvement  Framework 


Performance  Objectives 

•  Improve  Predictability, 
Consistency  &  Quality  of 
Service 

•  Increase  Productivity  & 
Reduce  Cycle  Time 

•  Maintain  &  Enhance  our 
Core  Competencies 

•  Improve  Customer 
Satisfaction 

•  Retain  &  Improve  our 
Competitive  Advantage 
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0  RDECOM  y  What  can  a  PIF  do  for  you? 


Business  Objective 

Process  Improvement  Objective 

Reduce  the  redundancies  in  the 
processes  deployed  across  organization 

Increase  efficiency  and  effectiveness  of  improvement  program 

Reduce  rework  and  quality  issues 

Re  enforce  training  and  develop  new  skills  and  capabilities 

Reduce  the  number  of  costly  “false-starts” 

Enable  achievement  of  growth 

Targeted  at  facilitating  high  value  activities: 

•  key  start-up  decisions 

•  appraisal  program  management 

•  ongoing  expert  process  improvement  advice 

Consolidate  audits  and  process 
compliance  reporting 

Implement  standard  processes 

Increase  portability  of  resources 

Transfer  lessons  learned 

Leverage  best  practices 

rf  2007  Award 
Recipient 
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Strategy 


Enterprise  Alignment  Strategy 


Strategic 

Intent 


Architecture 
& 

Design 


Vision  &  Mission 

-  Strategic  considerations 

-  Align  mission  needs 

-  Align  business  goals 

-  Select  improvement  tools  & 
techniques 

Organizational  Processes 

-  Comprised  of  elements  from 
underlying  models,  standards, 
quality  frameworks 

-  Dynamic  process  architecture  c 

-  Robust  organizational  processes 


Implementation 


OSSP 


Implementation 

-  Efficient  &  effective  performance 
improvement  infrastructure  0 

-  Process  asset  library  &  O 

measurement  repository 

-  Benchmarking  appraisals 

-  Compliance  audits 


Malcolm  Baldrige 

r  National 
Quality 
jAward 


Enterprise 
Organizational 
Set  of 
Standard 
Processes 
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IMPROVEMENT  MODELS .  STANDARDS .  PAG 

TOOLKIT  QUALITY  EXCELLENCE 


US  ARMY 


0  RDECOM 


Performance  Improvement 
Framework  (Integrated) 


OM 


BALDRIGE  Organizational  Excellence  Benchmark 


Baldrige  Criteria 


Organizational  Excellence  Measures 


CMMI-DEV  -  VI  .2/VI  .3 
CMMI-SVC  -  VI  .2/VI  .3 
CMMI-ACQ  -  VI  .2/VI  .3 
ISO  9001 
People  CMM 


j 


Statistical 
Process  Control 


AF 

>PI 

LY 

AF 

>PI 

LY 

Lean  Six  Sigma 
(Investment  Focus) 

Lean  Six  Sigma 
(Project  Level  Focus) 

Benchmarks,  Models,  Standards, 
Toolsets 


Resolve  Investment  Redundancies  & 
Design  Solutions  to 
Bridge  ‘‘Improvement  Gaps ” 


Sources 


Best  Practice 


Identify  Redundancies  &  Design 
Solutions  to  Bridge 
“ Project  Process  Gaps ” 


Tailored  Implementation 


0  RDECOMlk  Model  Steps  Completed 


Created  the  Armament  SEC  OSP  in  Model  Wizard  (policies,  project 
procedures  and  organization  procedures) 

❖  Built  a  model  map  between  CMMI-DEV  vl  .2  and  Armament  SEC  OSP 


Validated  the  above  mapping  CMMI-DEV  vl  .2  to  Armament  SEC  mapping 


Created  a  model  using  the  existing  PA 
Audit  Checklist 


Armament  SEC 
OSSP 


♦  Created  an  appraisal  instance  within  the  tool 
using  the  CMMI-DEV  vl.2,  Armament  SEC  OSP, 
ISO  9001-2000,  ISF  (Integrated  System  Framework) 

♦  Established  an  approach  to  record  audits 
using  Appraisal  Wizard  with  the  above  models 
and  model  maps 


4# 


v\  v  ai  u 


2007  Award 
Recipient 
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Model  and  Mapping  Steps 
Completed 


om 


4SW 


Created  the  Benet  Process  model  in  Model  Wizard  using  the  Benet  QMS 

♦  Created  a  model  map  between  the  Benet  QMS  and  ISO  9001-2000 

Using  an  appraisal  instance  with  Armament  SEC  OSP,  Benet  Process 
model,  CMMI-DEV  vl.2,  ISF  and  ISO  9001-2000  extrapolated  a  map  between 
the  Armament  SEC  OSP  and  Benet  QMS 


Added  the  Benet  QSPs  into  the  Benet 
Process  model 


rr  2007  Award 
Recipient 


Benet 
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0  RDECOM  ^  Model  and  Mapping  Steps 

Completed  (continued) 


om 


'Af/j 


4SW 


Added  the  Quality  System  Procedures  into  the  Quality  Management 
System  model  in  Model  Wizard  for  Benet  Labs  implementation 

♦  Split  up  the  Benet  QSPs  and  entered  them  into  MS  Excel 

♦  Imported  these  into  the  existing  QMS/QSP  Model 


❖ 


Included  the  QSPs  in  the  existing  ISO  9001-2000  map 
♦  Began  mapping  the  relationships  between  the  QSPs  and  ISO 
9001 :2000,  focusing  only  on  the  strong  relationships  versus  all 
relationships. 

Need  to  validate  this  mapping  with  the  rj 

Benet  QSP  author  team. 


❖ 
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Recipient 


Benet 

QMS/QSP 
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Model  Mapping  Tool 


m 


Map  Bements  (MM007)  -  CMMI  vl.2  ==>  Armament  SEC  OSP  *031104 


Tree  View  Find  Map 


0-  ■-£  CMMI -DEV,  VI.  Z  Staged 
E!  Maturity  Level  Z 

Requirements  Management 
B  ©  REQM  SG  1 


}  REQM  SP  1.2 
\  REQM  SP  1.3 
}  REQM  SP  1.4 
\  REQM  SP  1.  5 
0  ©)  REQM  GG  Z 

REQM  GP  2,1 
}  REQM  GP  Z.2 
}  REQM  GP  2.3 
}  REQM  GP  Z.4 
}  REQM  GP  2.5 
\  REQM  GP  Z.6 
}  REQM  GP  Z.7 
\  REQM  GP  Z.S 
\  REQM  GP  2.9 
\  REQM  GP  2  10 
0  ©)  REQM  GG  3 

REQM  GP  3.1 
^  REQM  GP  3.2 
0}  Project  Planning 

S  ©F 


"  © 


«  £ 


Model  Element 


Cora  men  its 


□  ^ 

Policy  PM -06-5 

□ 

Policy  PM -06 -5a 

□  <5 

Policy  PM -06 -6b 

Q  ^ 

Policy  PM-06-6C 

□ 

Policy  PM-06-6d 

□ 

Policy  PM-Q6-6e 

□ 

Policy  PM -06-6 f 

Q  'V 

Policy  PM  -Q6-6g 

□  « 

Policy  PM-06-6h 

□  V 

Policy  PM-Q6-6i 

□  ^ 

Policy  PM-06-6J 

□  ^ 

Policy  PM-06-6k 

□ 

Policy  PM -06-61 

□  ^ 

Policy  PM -06-7 

□ 

Policy  PM -06 -Appendix  A 

-  ^  Policy  RM-05,  Requirements  Management 

□ 

Policy  RM-05-5 

► 

a  ■>, 

0 

El  <& 

Policy  RM-Q5-6b 

0 

□  ^ 

Policy  RM-05-6C 

□  ^ 

Policy  RM-05-6d 

Q 

Policy  RM-05-6e 

□  <s> 

Policy  RM-Q5-6f 

□  ^ 

Policy  RM-Q5-6g 

□  <5 

Policy  RM-05-6h 

a  sV 

Policy  RM-05-6i 

0 

i . 

Policy  RM-05-6J 

□  ^ 

Policy  RM-05-6k 

□ 

Policy  RM -05-61 

Q  ^ 

Policy  RM-05 -6m 

Q 

Policy  RM-05-6n 

□  ^ 

Policy  RM-05-7 

n  *.! 

Policv  RM-0  5  -Acoendix  A 

1 

|  Save  | 


|  2^S  Cancel 


SP  1.1  Obtain  an  Understanding  of  Requirements 

Dst’eiop  on  underspent  ding  with  the  req  uirements 
providers  on  the  meaning  of  the  requirements. 

As  the  project  matures  and  requirements  are  derived,  all 
activities  or  disciplines  will  receive  requirements.  To  avoid 
requirements  creep,  criteria  are  established  to  designate 
appropriate  channels,  or  official  sources,  from  which  to 
receive  requirements.  The  receiving  activities  conduct 
analyses  of  the  requirements  with  the  requirements 
provider  to  ensure  that  a  compatible,  shared  understanding 
is  reached  on  the  meaning  of  the  requirements.  The  result 


0 


Element  T ext  j"  Mapped  Elements  for  REQM  SP1.1  J 


AH  projects  will  have  their  requirements  (technical  and  non-technical)  documented  in  a  RSL  (or  equivalent),  understood, 
analyzed,  ace  epted,  and  b  as  elined. 


Mm 
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^Ward 
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Reporting  Steps 
Completed 


♦  Built  QMS  to  OSP  report 


♦  Used  the  ISF  as  the  mechanism  to  resolve  how  the  QMS/QSP  related  to 
Armament  SEC  OSP. 


❖  Created  the  QMS/QSP  to  OSP  map  by  extrapolating  a  map  using  ISF. 

A  report  was  completed  from  the  set  of  maps  we  created  using  ISF  as 
the  connector  between  ISO  and  CMMI. 


❖  Initial  observation  gave  an  indication  that  we  could  potentially  have 
problem  with  the  CMMI  generic  practices  due  to  the  way  they  are 
accounted  for  within  ISF. 


❖ 


rr  2007  Award 
Recipient 


We  later  discovered  that  if  the  specific  practices 
are  connected  to  the  appropriate  CPP 
objectives  then  the  necessary  connections 
will  still  be  made  between  the  CMMI  and  ISO. 


Armament  SEC 
OSSP 


CMMI 


*CPP  =  Critical  Process  Performance 
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Reporting  Steps 
Completed  (continued) 


^  Built  OSP  to  QMS  report 
^  Built  OSP  to  ISO  9001-2000  report 
A  Built  QMS/QSP  to  CMMI-DEV  report 
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0  RDEcnn/T) ►  Preliminary  Conclusions 


Areas  of  the  Armament  SEC  Organizational  Standard  Software 
Process  not  explicitly  addressed  or  emphasized  sufficiently  in  Benet 
Quality  Management  System  or  Quality  System  Procedures 


❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 

❖ 
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CP002,CP003  -  Specific  aspects  of  peer  review  related  to  policy 
and  process  related  work  products 
CP005  -  Lessons  Learned 

CP006  -  Identify  and  Obtain  Project  Specific  Training 
CP  CM002  -  Processing  and  Handling  of  PCRs 
CPI  03  -  Risk  Management 
CPI  08  -  Formal  decision  making  process  (DAR) 

CP1 10  -  Requirements  Development  -  operational  concepts  and 

scenarios,  requirements  baselines 

CP1 14  -  Document/assess/control  interfaces 

CP1 15  -  Make/Buy/Reuse  Specified  Products 

CP1 16  -  Product  Integration 

CP1 1 8  -  Capture  of  validation  work  products 

CP1 19  -  Transition  and  sustainment  of  products 
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Preliminary  Conclusions 
(continued) 


Areas  of  the  Benet  Quality  Management  System  or  Quality  System 
Procedures  not  explicitly  addressed  or  emphasized  sufficiently  in 
Armament  SEC  Organizational  Standard  Software  Process 


o 

QSP  4.2-1  - 

o 

System 

QSP  4.2-2  - 

o 

QSP  4.2-3  - 

<> 

QSP  4.2-4  - 
Files 

o 

QSP  4.2-5  - 

o 

QSP  4.2-6  - 
Amendment 

o 

QSP  6.2-1  - 

<> 

QSP  7.1-1  - 

o 

QSP  7.2-1  - 

o 

Proposals 
QSP  7.2-2  - 

w  ♦ 

Malcolm  naldrijjc 

W  National 
F  Quality 
Toward 

QSP  7.5-1  - 

Preparation  &  Control  of  Quality  Management 

Control  of  External  Documents  and  Data 

Control  of  Product  Drawings 

Backup  Restoration  Procedure  for  Electronic  Data 

Record  Maintenance 

Quality  Management  System  Manual  Distribution  & 

Competence  Awareness  &  Training 

Product  Realization  Planning 

Review  of  Customer  Requirements  &  Estimates 

Customer  Communication  &  Satisfaction 
Process  Control 
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Preliminary  Conclusions 
^ continued 


❖  Areas  of  the  Benet  Quality  Management  System  or  Quality  System 
Procedures  not  explicitly  addressed  or  emphasized  sufficiently  in 
Armament  SEC  Organizational  Standard  Software  Process 


o  QSP  7.5-2  -  Preventive  Maintenance 

❖  QSP  7.5-3  -  Product  Identification  and  Traceability 

^  QSP  7.5-4  -  Verification  and  Control  of  Customer  Supplied 
Product 

❖  QSP  7.5-5  -  Handling  Storage  Packaging  Preservation 
Distribution  and  Delivery  of  Products  and  Materials 

QSP  7.6-1  -  Control  of  Monitoring  and  Measuring  Devices 


rr  2007  Award 
Recipient 
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Preliminary  Conclusions 
(continued) 


*>  Looking  at  the  following  O  and  CMMI  best  practice 
implementations 


<> 

QSP  8.1-1 

o 

QSP  8.2-1 

o 

QSP  8.2-2 

o 

QSP  8.3-1 

o 

QSP  8.5-1 

o 

QSP  8.5-2 

o 

QSP  8.5-3 

f  Malcolm  Baldrige 

National 

Measurement  Analysis  &  Improvement 
Internal  Audits 

Monitoring  &  Measurement  of  Product 
Nonconforming  Product  Review  and  Disposition 
Corrective  Action 
Customer  Complaint 
Preventive  Action 
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Takeaways 


o  Ensure  management  buy-in 

❖  Potential  for  multi-model  appraisals 

❖  Incorporate  strengths  of  multiple  models  and  standards 

❖  Continuous  feedback  and  Lessons  Learned  to 
organization  throughout  steps 
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Questions 


rf  2007  Award 
Recipient 
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Nathaniel  Becker 

Chief,  Process  &  PM  Engineering  Support 
Branch, 

Armament  Software  Engineering  Center 

nathaniel.becker@us.armv.mil 

973-724-6282 
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Project-Driven  Process  Improvement: 


A  Large-Scale  Lean  Six  Sigma 
Deployment  Across  Small-  and 
Medium-Sized  Organisations 

■ 

RPTT5  Software  Engineering 
Competence  Center 

Dr  Radouane  Oudrhiri  Dr  Amr  Kamel 

radouane@systonomy.com  aakamel@itida.gov.eg 
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11th  Annual  CMMI  Technology  Conference  and  User  Group 
14-17  November  2011  -  Denver,  CO 


BS  Software  Engineering 
P  Competence  Center 


AGENDA 


■ 


* 


systonomy 


♦  The  Egyptian  IT  landscape 

■  Key  Facts 

.  The  Role  of  ITIDA  -  SECC 

#  The  Lean  Six  Sigma  Initiative  for  IT  and  Software 

■  Objectives 

■  Participating  Organisations  -  Companies  profile 

■  Types  of  Projects 

■  Deployment  approach 

■  Preliminary  Results 

#  Challenges  and  Lessons  Learned 

♦  Q&A 
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Objectives  of  the  presentation 


$ 


systoroTiy 


#  Share  with  you  the  experience  of  Three-Party  setup  for 
deploying  a  Lean  Six  Sigma  Initiative  in  Egypt,  in  terms  of: 


■  Implementation  and  delivery  model 

■  Organisations  involved 

■  Challenges 

■  Commonalities  and  differences 

■  Key  lessons  learned 


\ 

©ida 

Developing 

IT  in  Egypt 

V 

IT  SERVICES  . ^ 

■ 

,  i  (  Software  Engineering 
? 4  \°  Competence  Center 

SOFTWARE 


systonomy 


flUfruNK:  -Mentor 

Graphics 


b.  »  4m3^NI  pl***^iU 
Systems  Engineering  of  Egypt 


ITW^RX 


GIZR 

SYSTEMS 
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"  Competence  Center 


Egypt’s  strategic  focus  areas 


* 


systonomy 


Ministry  of  Communication 
&  Information  Technology 


®ida 

Develcona 


enveloping 
IT  in  Egypt 
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The  Egyptian  IT  landscape 


* 


systonomy 


2011  Ranking 

1  |ndia 


2  China 


3  Malaysia 


l  4  Egypt 


5  Indonesia 


0  Mexico 


7  Thailand 


Q  Philippines 


15  UAE 


24  Poland 


35  Czech  Rep 


37  Morocco 


2009  Ranking  2009  vs.  2011 


1 

2 

3 

6 


O 


Financial  attractiveness 
Skills  availability 
Business  environment 


5 

— 

rEgypt  has  ^ 

11 

t 

improved  by  9 
positions  in  4 

4 

^years.  j 

7 

\ 

(9 

t 

38 

t 

A.T  Kearney  tanks 

33 

Egypt  number  1 
Global  Service 

30 

\  1 

Delivery  location  in 

the  EMEA  region 

A.T.  Kearney  Global  Services  Location  Index  2011 
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Competence  Center 


■ 


&  Mission 


systonomy 


♦  Vision: 

■  To  enhance  the  quality,  efficiency  and  level  of  innovation  of  the  ICT  companies 
and  improve  their  global  competitiveness 

♦  Mission: 

■  Provide  services  improving  technical  competence  and  internal  capabilities 


Profile: 

Sole  Certified  Lead  Appraisers  for  CMMI  in  the  Middle  East 

Sole  Examination  Institute/Certification  Body  for  the  ISTQB®  certificates  in 
Egypt 

Highest  number  of  IT  experts  under  one  roof  in  the  Middle  East 
Largest  Sponsor  of  IT  technical  capacity  building  in  Egypt 
Large  array  of  experiences  with  Multilanguage  capabilities 
Serving  more  than  450  customers 
Offering  a  portfolio  of  12  services 
More  than  8  international  cooperation  &  alliances 


SEIPartner 


Accredited 
Training  Partner 


ESTB 

I J  Egyptian  Software  Testing  Board 

ISVQCenteF 
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P  Competence  Center 


SECC  -  Suite  of  Services 


Capability  Maturity  Model  Integration  for  Development  (CMMI-DEV) 
Software  Process  Improvement  Guide  (SPIG) 

Personal  Software  Process/Team  Software  Process  (PSP/TSP) 
Embedded  Software 
Software  Reuse 
Software  Testing 

Information  Technology  Infrastructure  Library  (ITIL) 

Capability  Maturity  Model  Integration  for  Services  (CMMI-SVC) 
Certification 
Training 
Agile 

Lean  Six  Sigma  for  IT  and  Software 


®ida 

Develoninn 


Developing 
IT  in  Egypt 


Services  are  sponsored  by  ITIDA  (through  corporate 
tax  revenues)  and  offered  to  Egyptian  IT  companies  at 

substantially  reduced  cost 
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Systonomy  Profile 


systonomy 


#  Founded  in  1999 

■  Headquarter  in  London 

■  Presence  in  Europe,  US  and  ASIA 

<#>  Dedicated  to  Quality  Engineering 

■  Lean  Six  Sigma  and  DFSS 

■  Leadership  Development 

■  Software  Process  Improvement 
(CMMI®,  ITIL,  PSP...) 

■  Strategic  Consulting 

■  IT  and  Software  Risk  Management 

■  Training  and  Coaching 

■  Hands-on  Project  Implementation 


systonomy 
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P  Competence  Center 


Our  Mission 


♦  Our  business  is  Software  Quality 
Engineering  and  Lean  Six  Sigma 
for  ICT  and  Software  Development 

♦  We  work  with  any  organisation 
where  Software  and  IT  is  a  critical 
part  of  the  business 

♦  We  help  our  client  to  accelerate 
their  investment  in  continuous 
improvement  in: 

■  Software  process  performance 

■  Software  product  quality 

■  ICT  services 


systonoTTiy 


SIEMENS  0  BOSCH 

#  BARCLAYS  B  gK; 


orange 


m 


GlaxoSmithKline 
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Competence  Center  The  programme  set-up  and  roles 

#  Three-party  setup  and  work  dynamics 

■  Knowledge  transfer 

■  Lean  Six  Sigma  for  IT  and  Software  Training 

■  Project  Coaching 

■  Companies  Project  implementations 


Management  and/or  hands-on 
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Deployment  approach 


* 
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♦  Entire  initiative  managed  on  a 
project  by  project  basis: 

■  Companies  selection  (based  on 
proposed  Lean  Six  Sigma  projects) 

■  Training  (aligned  with  projects 
calendar) 

■  Coaching  (hands-on  project  work) 

■  Certification  (based  on  project 
results  and  not  only  theoretical 
knowledge) 
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Organisations  and  Projects  Selection 


systonomy 
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Organisations  and  projects  selection 


Company  fitness  Company  Name 


Criteria 

Importance 

Problem  Awareness 

1.0 

Problem:  Fitness  for  ITIDA/Pilot 

1.0 

Individual  Commitement 

1.0 

Individual  Availability 

1.0 

Individual  Fit  (as  a  Team) 

1.0 

Organisational  Sponsorship 

1.0 

Long  Term  Sustainability 

1.0 

Track  record  -  Improvement  Initiatives 

1.0 

Relationship  with  SECC 

1.0 

Total 

% 

1^ 

CO 

o> 

o 

o 

o 

> 

> 

> 

c 

c 

c 

(0 

(0 

(0 

a 

a 

a 

E 

E 

E 

o 

o 

o 

o 

o 

o 

M 

M 

M 

|  M 

l  i 

CM 


V 


Criteria 

Weight 

Problem  Awareness 

W1 

Problem:  Fitness  with  SECC/Pilot 

W2 

Individual  Commitement 

W3 

Individual  Availability 

W4 

Team  Fitness 

W5 

Organisational  Sponsorship 

W6 

Long  Term  Sustainability 

W7 

Track  record  -  Improvement  Initiatives 

W8 

Relationship  with  SECC 

W9 

BS  Software  Engineering 
P  Competence  Center 


■ 


Lean  Six  Sigma  Initiative  -  Participants 


systonomy 


#  A  broad  and  diverse  group  of  organisations,  reflecting  the 
composition  of  the  local  market 


■  Software  Development 

■  IT  Services 


LINK 


-Mentor 

Graphics’ 


ITW^RX 


■  Business  Process  Outsourcing 

■  Government  Organisations 


IT  SERVIC 


ri  i  ilj  l /h 1 1  n  L""k ■  1 1 

Si^lnnrp  Rng[n^^in(]  nf  EfiP1 
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Competence  Center  Selected  Companies  $  systoriomy 

♦  The  majority  of  selected  companies  are  mid-sized  or  SME’s 
companies 

■  Local  and  international  presence 


#  Engaged  in  Projects  Improvement  initiatives  in  the  past: 

■  CMMI 

■  PSP/TSP 

■  ITIL 

-  COPC 

#■  Original  Lean  Six  Sigma  motivations 

■  Process  Excellence  (attraction  of  international  clients) 

■  Individual  motivation  (Certification) 

■  Trained  on  “classical”  Six  Sigma  but  don’t  know  how  to  apply  it  to  IT/Software 

■  Measurement.. .estimation... 

■  A  mean  to  achieve  high  maturity  levels 
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IT  and  software  culture  and  challenges 


systonotTiy 


IT  Organisations 

<#■  Culture  &  Challenges 

■  ITIL  Culture 

■  Closer  to  the  customer 

■  More  tangible  benefits 

■  Straightforward  improvement 

■  Have  useful  data  or  were 
able  to  generate  data  easily 


Software  Organisations 

#  Culture  &  Challenges 

■  Tend  to  think  through 
compliance  to  a  maturity  model 

■  Struggle  with  problem  definition 

■  More  technical  than  business 
oriented 

■  More  complex 

■  Heavily  rely  on  project 
development  schedule 

■  More  invasive  and  disruptive 

■  Protective  attitude 

■  Various  levels  of  maturity 
(CMMI  LI  to  L5) 

■  Some  have  huge  volumes  of 
data,  but  rarely  useful 
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Problem  Solving  Methodology  -  a 

rigorous  approach  (Reminder) 


systonomy 


Six  Sigma  Integrates  Project  &  Change  Management  and  rooted 
into  Empirical  and  Experimental  Software  Engineering  studies 
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B  Software  Engineering  ■  ■  I  ■■ 

£  Competence  Center  billing  de  IVety 


♦  Black  Belt  Training  (4  weeks) 

■  13  BB’s  from  SECC  and  companies 

♦  Green  Belt  Training  (2  weeks) 

■  40  GB’s  from  companies 


#  Importance  of  Theory  and 
Practice 

■  No  theory... No  Learning  (Deming) 

#  Each  delegate  must  have  a 
project 

■  No  project...  No  training 


♦  Lean  Six  Sigma  does  not 
mean  “Light”  Six  Sigma 

#  Six  Sigma  is  NOT  just  about 
Statistics 


Key  comments 

■  Participants  surprised  by  the 

importance/emphasis  put  on  problem 
formulation  and  “soft”  skills... but 
recognised  the  value  later 

♦  “The  soft  stuff  is  the  hard  stuff 

♦  Some  people  want  to  jump  on  statistics 


■  Some  participants  previously  trained 
in  “Classical”  LSS  realised  the  major 
differences  with  IT  and  Software 


■  Teaching  statistics  remains  a  difficult 
task 

♦  Use  of  visual  exploratory  tools 

♦  “Unlearn”  some  misconceptions 
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Coaching  activities 


systonofTiy 


4>  Coaching  delivered  through  focused  and  intensive  sessions 

■  Often  2  coaches,  to  get  different  views  on  the  problem 

■  Complement  to  training  by  enabling  effective  knowledge  transfer 

■  Practical  knowledge  in  Software  Engineering  and  Lean  Six  Sigma 
methods 

♦  People  expect  the  coach  to  help  solve  the  problem  and  be  familiar  with  IT 
and  Software 

■  The  coach  should  not  do  things  instead  of  the  team,  but  bring  people 
to  do  it  by  themselves 

♦  Provocative  and  naive  questioning 


♦  Coach  SECC  consultants  to  become  coaches 

■  Train-actionSM  (from  French  Formaction) 


#  Minimum  of  3-5  days  coaching  per  phase 
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Projects  and  (initial)  Problem  areas  identification 


♦  Selection  Criteria 

■  Strategic  Issues 

■  Type  of  Problems 

■  Expected  Return  on  Investment 

■  Measurability 

■  Organisational  Issues 

■  Project  Issues 


♦  Pragmatism 

■  Start  from  the  areas  closer  to 
the  final  customer 

■  Stop  the  bleeding  first! 


...  Start  Downstream 


◄ -  Upstream  -  Downstream 

-  - ► 
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^sssssss?  it  and  software  project  types  •* 

■#  Projects  have  been  selected  and  refined  during  the  initial 
phase  of  the  DMAIC  Six  Sigma  roadmap 


SW  Companies 

IT  Services  Companies 

Project  Type 

Nb 

Project  Type 

Nb 

Test  Effectiveaess. 

2 

Incident  Management 

2 

^Code  defects  containment 

jf;; 

Capacity  Planning 

1 

III  defects  reduction 

1 

Delivery  Cycle  Time 

1 

Regression  testing  -False 
defects 

1 

Service  Requirements 
effectiveness  (Requirement 
Defects) 

1 

Project  management  overhead 

1 

Acceptance  test  cycle  time 

1 

> 


A  common  problem  to  many  companies 
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Lean  Six  Sigma  for  SW  Process 
improvements  Experimental  Projects 


systonomy 


♦  Development  projects  are  objects  of  studies  and  implementation  for 
the  Lean  Six  Sigma  Projects. 


DEFINE  >  MEASURE  > ANALYSE  >  IMPROVE  ' 

>  CONTROL  ^ 

Six  Sigma  Project 

_ * 

Lean  Six  Sigma  projects  must  be  synchronised  with  development  projects 
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Sigma  SW  Projects  -  findings 
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DEFINE 


Initial  problems  often  expressed  as  “Non  compliance”  or  absence 
of  process 


“CMMI  excuse” 


Why  is  this  problem? 

BTW:  processes  always  exist... 
Who  tells  you  that  if  you  follow  that 
process  you  will  perform  better? 


Shift  the  thinking  towards  “Pain”  related  to: 

■  Effectiveness  first  ->  Defects 

♦  Late  phases  first  and  go  up  the  process 

■  Efficiency  second  ->  High  Effort,  Waste 


Very  difficult  to  admit  for 
CMMI  L2  and  L3,  that  we 
have  quality  problems 


♦  Again  the  CMMI  excuse,  CMMi  asks  for  lot  of  overhead  activities  and/or 
documentation  (some  companies  have  over  35%  management  overhead) 


> 


#•  Shift  the  thinking  about  Cost  of  Poor  Quality 

■  Pain  means  cost 


Define  phase  ranged  from  6  to  15  weeks:  paradigm  shift  from  compliance 
oriented  improvement  to  performance  oriented  improvement...  Shock  therapy 


Business  Case 

Management: 

...  Now  we  are  talking! 
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competence  center  |_ean  six  sigma  SW  Projects  -  findings  1/3 


MEASURE 


#  Most  existing  measures  are  not  useful... 


■  Again  “CMMI  excuse” 

■  KP I mania  syndrome 


So  how  this  is  related  to  our  problem? 
Anyway,  what  can  I  do  (or  get)  from 


♦  KPI’s  are  collected  like  Pokemons  these  measures?” 

.  3rd  order  measure  vs  1st  order  “Why  do  you  collect  them?” _ 

♦  Complicated  is  Beautiful 

■  Time  reporting  system 

♦  One  of  the  most  unreliable  and  distorted  source  of  data 

♦  “We  do  not  have  measures” 

■  You  often  have  (or  can  easily  get)  more  measures  than  you  think! 

♦  Focus  on  First  Order  Measures  Defects 

■  Defect  Categorisation 

■  Cost  of  defect 

It  is  not  about  collecting  measures  for  compliance ,  but  finding  the  key  factors 
(X’s)  that  influence  our  problem  (Ys). 

MEASURE  PHASE  ranged  from  8  to  15  weeks. 
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Lean  Six  Sigma  SW  Projects  -  findings  2/3 


■ 


>systonom\ 


MEASURE  ' 


#■  9  Six  Sigma  projects  analysed  between 
590  to  1070  defects  (Code  defect 
Containment,  Testing,  Regression  test) 


■  Analysis  conducted  as  a  workshop 
gathering  people  from  multiple 
perspectives 


#■  To  the  surprise  of  all  it  only  took  max  2 
hours  to  analyse  100  defects  ->  2-3 
days  to  analyse  a  large  enough  sample 
of  defects! 


■  Reliability  and  agreement  on  the  defect 
assessment  by  design 

■  First  Stage:  Operational  definitions  provided 

■  “Calibration”  of  the  team 

■  Adaptation  and  refinement  of 
Operational  definition 

■  Second  stage:  more  reliable  and  quicker 


♦  Pe°P|e  d0  have  a  memory for  defects!  Defects  Characterisation 

Provides  the  basis  for  the 

#  Defects  analysed  on  multiple  organisation  measurement  system 

dimensions 

■  Type  of  defects 

■  Complexity  -  T-shirt  sizing  (XL,  L,  H,  XH) 

■  Effort  to  fix 
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Lean  Six  Sigma  SW  Projects  -  findings  3/3 
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ANALYSE 


♦  Typology  of  defects  help  refining  the  problem,  will  drive  the  projects 
and  the  solution 


♦  Root  cause  Analysis 

depends  on  the  type  of  defects 


Looking  for  the  Pareto  effects 


♦  Analyse  is  not  only  about  statistics 

■  People  tend  to  jump  on  the  statistical  analysis  tools 

■  System  and  structural  tools  are  as  much  important  as  the  statistical  tools  -> 
provides  context  to  the  data 

ANALYSE  PHASE  ranged  from  4  to  8  weeks. 
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Six  Sigma  SW  Projects  Example  of 
experimentation 


— 


* 
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ANALYSE 


> 


DEFINE  ^  ^  MEASURE^) 


7>  ANALYSE^ 


^ IMPROVE^  ^  CONTROL^ 


Min  50  points  Size 

Pull  planning  rather  than  push  Team  size 

(e.g.  Inspection  every  2  weeks  per  project)  Process  steps 

Effort/Time 
per  step 


V 


Experiment  Parameters: 

Size 

Team  size 
Process  steps 


Effort 

etc 


/  \ 

optimise 

V _ 
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Inspection 
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Unit  Test 

3 
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CD 

Code  Analyse 
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Competence  Center  Six  Sigma  SW  Projects  -  findings 

z^m^IMPROVE  y 

♦  Pilot  projects 

■  Code  Inspections 

■  Ul  Defect  Containment 

■  Testing  Cycles 

♦  Software  Projects  showing  slower  paced  improvement 

■  More  complex 

■  Cultural  barriers 

■  Economic  slowdown  due  to  recent  events 

♦  ...but  the  initial  analysis  of  the  implemented  improvements  data 
shows  potential  for  remarkable  increases  in  Defect  containment 

♦  Financial  Benefits  so  far:  2.5:1  to  10:1 

♦  Cultural  resistance  is  the  key  challenge,  but  cultural  change  is 
always  possible  (and  the  Egyptians  know  this  very  well...) 
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e.  e.  1T  .  ,  f.  ..  < 

L:"  ■  - . -  Six  Sigma  IT  projects  -  findings  / 

■#  Problem  Space  more  straightforward 

■  Delays  and  Cycle  time  (Lean  focus) 

#  Highly  visible  to  the  Customer 
(“louder”  VOC) 

■  Shift  from  service  denial  to  true  a  service  provider 


Histogram  of  ResolutionTime  -  Banking 


#  Performance  measures  already  available 

■  SLA  adherence,  response  times,  Non  value  added  time 

■  Tangible  COPQ  (rework,  duplication,  penalties..) 

#  Process  more  directly  observable 

#  “Continuous”  flow  of  data  (incidents) 

Compared  to  Software  organisations ,  IT  services  company  already  have  a 
culture  of  process  excellence ,  commitment  to  customer  and  measurement 
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Bi  Software  Engineering  ■  g 

f  Competence  Center  §|X  SigHTId  IT  prOjCCtS 


#  Process  Value  Analysis  (Lean) 


♦  Root  Causes 


■  Incorrect  incident  classification  / 
prioritisation 

■  Non  value  added  time 

■  Capacity  Planning 


#  Importance  of  Process  and 
Data  view 

■  Huge  volumes  of  data,  require 
context  and  interpretation 


Approach 


systonotTiy 


Boxplot  of  RestorationTime 
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Competence  Center  Six  Sigma  IT  Projects  -  findings  «* 

i^^^IMPROVE  y 

#  Pilot  projects 

■  Incident  Management 

■  Capacity  and  resources  planning 

■  Delivery  and  procurement  cycles 

#  IT  projects  showing  significant  reductions  in  terms  of  cycle 
time,  improved  capacity  planning  and  process  governance 

■  Lean  process  improvement 

■  Six  Sigma  for  establishing  ongoing  monitoring  and  controls 

■  Process  harmonisation 

■  Customer  involvement,  focus  on  delivering  value 

#  Straightforward  improvement  with  quicker  tangible  benefits 

#  Huge  Financial  Benefits  so  far  ROI:  7.4:1  to  17:1 
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BS  Software  Engineering 
P  Competence  Center 


Change  is  possible... 


From 

To 

(Managers)  decisions  are  rarely  questioned 

Everything  can  and  will  be  questioned  openly 

Decisions  are  based  on  experience,  influence, 
power  relations  and  gut-feelings 

Decisions  are  based  on  in-depth  knowledge  of 
structural  and  quantitative  characteristics  of  the 
system 

Quality  is  viewed  as  compliance,  controls, 
checklists  and  extra  processes 

Quality  is  seen  as  a  business  enabler  and  a  way 
of  working 

Buzzwords  and  tecchie  jargon  are  used 
extensively  to  justify  improvement  efforts 

Clarity  and  Operational  Definitions  are  used  to 
build  consensus  and  drive  change  initiatives 

Solutions  are  often  quick  fixes 

Problems  are  tackled  at  the  level  of  the  root 

causes 

Improvement  (if  any)  is  “by  the  book” 

Improvement  is  “by  experiment  “ 
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Software  Engineering 
Competence  Center 


Lessons  Learned 


■ 


#  Large  scale  deployments  can  accelerate  the  investment  in 
capacity  building 

#  They  do  however  present  specific  challenges 

■  Managing  multiple  projects  in  the  same  process  area,  but  with 
different  organisations 

■  Synergies  across  projects,  knowledge  sharing  across  organisations 

#  Cultural  change  and  resistance 

■  Use  projects  to  drive  change  (tangible  results) 

■  The  exploratory  part  and  problem  definition  play  a  major  role  in  raising 
awareness  and  getting  the  “questioning  habit ’ 

■#  Data  is  A-politycal...  Data  makes  a  difference 

■  But  I  don’t  believe  your  data...  I  might  believe  my  own  data! 

■  First  order  Measurement  come  first  (keep  it  simple) 
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5  Software  Engineering 
P  Competence  Center 


Questions? 


systonomy 


Thank  You! 
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A  Process  Performance 
Models  Case  Study 

Based  on  sample  (from)  450  project 

feasibility  checks 

and 


Presented  with  practical  usage  and 
implementation  tips 


Agenda  and  Topics 


Opening 

Organizational  Background  and  Process  ROI 

•  Project  Idea  and  Proposal  Preposition  Development 

•  Quality  Audits  and  Progress  Check  Calibration 

•  Call  and  Incident  Center  Performance 

Case  Studies  and  High  Level  Process 

•  Project  Idea  and  Proposal  Preposition  Development 

•  Quality  Audits  and  Progress  Check  Calibration 

•  Call  and  Incident  Center  Performance 

Main  Questions  for  High  Maturity  Process 
Improvement 

Pilot  Lessoned  Learned 


A  set  of  interrelated  activities,  which 
transform  inputs  into  outputs,  to  achieve 
given  purpose. 


rocess  Control 


Key  Questions 

•  What  is  the  current  performance? 

•  Is  this  value  "good”? 

•  Is  it  changing? 

•  How  can  I  make  the  value  “better”? 

Candidate  Attributes 

•  Definition  (completeness,  compatibility) 

•  Usage  (compliance,  consistency) 

•  Stability  (repeatability,  variability) 

•  Effectiveness  (capability) 

•  Efficiency  (productivity,  affordability) 

•  Predictive  Ability  (accuracy,  effects  of  tailoring  and 
improvements) 


Some  Examples 


Goal 


Measure 


The  number  of  process  elements  added,  changed,  and  deleted  during 
tailoring. 

Number  of  discrepancy  reports  generated  by  Quality  Assurance  audits 

The  number  of  process  elements  changed  within  a  specified  time 
interval. 

Product  quality 

Defect  leakage  to  subsequent  phases 
Productivity  (or  production  coefficient) 

Rework  as  a  fraction  of  total  effort 

Probability  distribution  for  an  estimated  quantity  or  related  population 
statistics 


Completeness 

Compliance 

Stability 

(volatility) 

Effectiveness 

Effectiveness 

Efficiency 

Efficiency 

Predictability 


Opening 


Typically  when  one  read  the  CMMI-SVC  he  may  think 
on  the  classic  service  provider  organization 

The  model  provides  guidance  for  the  application  of 
CMMI  best  practices  by  the  service  provider  organization. 

Best  practices  in  the  model  focus  on  activities  for 
providing  quality  services  to  the  customer  and  end  users. 

In  this  presentation  the  ‘services’  is  a  project  feasibility 
checks  provided  by  a  dedicated  group 
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ganizational  Background  and  Process  ROI 


roject  Idea  and  Proposal  Preposition 
Development 

If  an  average  developer  day  cost  is  -7000 
The  total  Program  effort  was  10220  day  (100%) 

The  testing  phase  was  1480  day  (14.5%) 

Defect  that  are  the  result  of  documentation  are  69%  of  all  defects 


If  we  will  assume  the  to  correct  69%  of  all  defects  will  take  around  40%  of  th< 
testing  duration;  ^  means  that: 

•  that  will  be  740  day 

•  With  the  overall  cost  of  5 1 8000 

•  However  to  add  100  review  days  in  the  static  tests  and  another  20  of  code 
inspection  will  end  with  the  cost  of  2100000 


And  still  we  have  saved  at  least  3080000  (440  days) 

Means  that  we  ware  able  to  reduce  4.5%  of  the  project  time 


ganizational  Background  and  Process  ROI 

Quality  Audits  and  Progress  Check 


Calibration 


As  for  today  most  of  major  industries  which  runs  and  mange  large  and 
complex  programs  need  to  comply  with  more  than  just  one  quality 
standards  in  many  disciplines  (e.g.  HW,  optics,  software)  use  large  groups 
of  internal  and  external  assessors  that  perform  implementation  checks, 
progress  checks,  readiness  reviews  and  formal  appraisals. 

These  communities  are  typically  composed  from  groups  of  very 
experienced  and  professional  individuals  that  have  the  best  knowledge  in 
their  professional  domain  but  not  necessarily  on  how  to  conduct  an 
efficient  and  effective  appraisal  which  provide  meaningful  results 

The  combination  of  the  effort  and  expected  resources  increase  the  risks  on 
qualification  of  auditors,  domain  knowledge,  and  calibration  of  results  and 
findings  effectiveness 


ganizational  Background  and  Process  ROI 

Quality  Audits  and  Progress  Check 


Calibration 


By  measuring  the  following  attributes,  we  were  able  to  increase  usability 
of  the  progress  checks  by  47%,  and  quality  of  deliverables  by  37% 

Role  based  profile  and  criteria 
Calibration  mechanism  and  criteria 
Evaluation  mechanism  and  criteria 
Leveling  the  different  quality  engineers  and  ‘auditors’ 

Flowing  specific  trainings  (on  different  levels)  as  personal  development  and 
qualification  criteria 

Listing  specific  performances  as  indicators  for  leveling  justifications 

Structuring  the  different  audits  and  reporting  guidelines  in  a  single  mandatory 
to  follow  process, 


Main  Steps  for  High  Maturity 
Process  Improvement 


During  our  analysis  and  planning,  we  were  able  to  identify 
improvement  targets  in  main  lifecycle  areas  such  as 

•  operations, 
information, 

•  governance, 
people 

organizational  structure, 
portfolios, 

•  project  execution, 
finance. 

And  as  in  core  process  that  are  critical  to  the  system  success  such  as 
stakeholder  management,  technical  interfaces  and  integration. 


Main  Steps  for  High  Maturity 
Process  Improvement 

the  result  of  this  observation  we  have  built  an  action  plan, 

Then  in  the  second  step  we  have  built  a  services  roadmap 

using  the  CMMI-SVC,  that  allow  companies  to  begin  the 
improvement  journey,  and  manage  the  transformation  to 
maturity  by  building  on  each  successive  step,  and 
ultimately  delivering  the  benefits  expected: 

•  service  reuse, 

•  improved  perception 

•  response  time, 

•  interoperability, 

•  business  agility. 

Service  performance  and  its  impact  on  the  organization 
governance  is  a  significant  part  of  that  journey 


Organizational  Background  and 

all  and  Incident  Center  Performance 

The  service  provider  provides  a  large  number  of  services 
to  its  customers,  which  are  mainly  departments  from  a 
sibling  organization. 

To  manage  the  communication  with  customers  regarding 
those  services,  the  department  has  implemented  helpdesk 
management  and  problem  management  processes. 

The  implementation  of  these  processes  has  been  based  on 
the  CMMI-SVC  with  elements  of  other  CMMIs  (for  the 
organization  maturity)  and  ITIL  (for  the  individuals’ 
education). 


Organizational  Background  and 

all  and  Incident  Center  Performance 


Program  Management  Office  is  used  to  guarantee  the 
continuity  of  services,  while  Analysts  Management  Team 
is  used  to  improve  the  level  of  service  in  the  future.  So, 
PMO  deals  with  requests,  whereas  Requirements 
Management  Team  is  concerned  with  solving  the 
challenges  that  cause  these  requests. 

The  goal  of  this  case  study  was  to  assess  the  quality  and 
performance  of  the  feasibility  checks  Management  process. 
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o.  ra  functions  <—  ^ 
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reiSO 


0)||S  Q) 

»— 1  ,Vak^“ 
Q  factor  Ur 

re  companies 
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«  firm 


compnses|w 

|  enterprise  o 

^  rectear,Y  .  fiil®!SLSS 


o 

o 


but  is  clueless 

about 


CMMI 
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P 
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K°uisq 


except  she's 
heard  it  creates 


0 

Requirements 

0 

Design 

□ 

Coding 

□ 

Testing 

□ 

Installation 

on 

schedule! 


on 

budget! 


happy 

customer! 


She  even  has  a 

new  boyfriend 


..with  abs! 


V 


Life  is  good 
for  Pam! 


One  day  Pam 
gets  an  e-mail 


Uh-oh 


Use  our  new  CMMI processes 


L 


*0f  course  we  know  there's  no  such 


Vij 


Deployment 


Do  this  (now) 

because 
we  said  so! 


I'll  look  at  that 
stuff  when  I 
have  time. 


Whatever.  i 

^  - _ 

O 

o 


Pam's 

CMMI® 

Rock-O-Meter 


5 


Then  one  day 

Pam  has  a 


Maybe 

(just  maybe) 

this  "CMMI®  stuff 

can  be  useful! 


Pam  wants  a... 


Te«  Plan 


She  consults  the  PAL* 


^process  asset  library 


...and  finds  templates  for  systems  like 

this 


...but  her  system  is  more  like  this 


She's 

forced 

to 

develop 
a  plan 


on 
her 
own 


Crap!  I  thought 
the  CMMI®  was 
supposed  to 
help  with  stuff 
like  this! 


Pam's 

CMMI® 

Rock-O-Meter 


3 


pays 
a  visit... 


Y  N 

□  or 

□  S' 


Blah  blah  blah  data 
management  plan. 

Blah  blah  blah 

stakeholder  involvemen 
plan. 

Blah  blah  blah 
blah  blah  blah. 


But  the  project  is  half  over! 


You  must  follow  the  process. 


I  have  no  time  for  this! 


You  must  follow  the  process. 


It  doesn't  help  my  project! 


You  must  follow  the  process. 


The 

rules  were  changed 

on  Pam 

mid-stream! 


She  works 

day. 


Addressing 

non-compliances 

instead  of 


SotheCMMI® 
really  is  just  about 

paperwork... 


...and  now 
I'm  behind 
schedule! 


Pam's 

CMMI® 

Rock-O-Meter 


o 


Pam  discovers 

appraisals 


We  need  you  to 
populate  an 

evidence  matrix 

for  our... 


SCAMPI* 


^Standard  CMMI®  Appraisal  Method  for  Process  Improvement 


OurVP  says 

you  have  to 


The  project's  new 
part-time  job, 


"populate 

evidence 

matrix"... 


takes  forever 


good  enough? 


EPG* 


^engineering  process  group 


try  again! 


The  project  is  now  so 

distracted 


® 


11 


byCMMI 
compliance 

that... 


## 


...the  team  is  burned  out 


...the  customer's  not  smiling  anymore 


...and  Pam's  boyfriend 
dumps  her! 


Meet  the 
new  Pam 


Why  did  Pam  become  a... 


Why  do  project  managers 


j.  la 

[J — :  hi  j  t* 

vr 

II  jfc  1 

-  j 

m3  1 

often... 


...theCMMI®? 


Not  because  of 


CMMI 

for  Development 

Guidelines 

for  Process 

Integration 

and  Product 
ivernent 


lmpro’ 


Third  Edition 


vlaryBcthChri^s 

Mike  Konrad^ 

Sandy  Shruro 


Q  Process  development 


...and  ftndstemplates  for  systems  like 

this 


She's 

forced 

to 

develop 


one 


,/ 

> 


on 

her 

own 


C  ©  ( 


evelopment 


rocess  deployment 


The  project's  new 

part-time  job, 


Q  Appraisals 


O  Process  development 
0  Process  deployment 
0  Appraisals 


We  could  go  on  and  on 

and  on 

and  on 
and  on 

about  each  of  these... 


Instead 

let's  focus 

on  a  few 

process 

improvement 

best 

practices... 


...that  a 
project 

manager 

might 

value 


QOO 


Process 

development 


Ensure  that  your 
process  developers 

appreciate 

the  model 


itJtfty//lT/Q  4/ 


Srmrer^  - - / 


'T/N&er 


Qtfatfr 


f 


this  is  appreciating  the  model 


this  is  appreciating  the  model 


this  is  appreciating  the  model 


rv 


Tr 


Involve  people 
that  do 

"real"  work 


like  her... 


...but  maybe  not  him 


A  Fi 


Use 

project  planning 

to  plan  your 
PI*  project 


^process  improvement 


Open  the  book 


k  ^  — rnmr| 


Of 


,u^Wn 
'‘rtud*  \ 


lucU  \q 
IC\  \\ue 
ed  vjQr\^ 


ivotoes 

olty  m 

e\  p\an  to 

,s  \N>" 


jAC^ 


k>co9f‘ 


^ccv^ 


*  ■>\°9''  t^'°'.rtsl  ?^o\ec 


yV.'^ 


M&* 


vVeeAc< 

?V3aS^  ?,o\^X^ 

19  \V>  ®£_  A  ^  ,0v^Ns 


_  4 


ftS 


Vi 


Go* 


\ 


Don't  estimate  by  pulling 
numbers  out  of... 


Develop  useful 
tailoring 
guidance 


*and  spend  my  time  figuring  out  how  to  manage 
my  project  instead  of  actually  managing  it! 


Large 

Project 

Plan 

Template 


>  $iooK 


Plan 

Template 


$iooK 


o 


Process 

deployment 


Don't  just 
announce 
the  existence 
of  a  book 


Unless  this  is  the 
reaction  you  want 


I'll  look  at  that 
stuff  when  I 
have  time. 


Whatever.  i 

^  - _ 

O 

o 


...is^Ja  deployment  plan 


Communicate 


Train 

■  i  dill 


Ensure  Access 

Learn 

rca i  i  i 


Schedule  Monitor 

r* i  im n uiaiiitai 


Guide 


senior  leadership 
support 


I 

communication 


training 

(of  those  impacted  by 
the  initiative) 


Source:  The  CMMI  Success  Factor  Survey,  ACME  Process  Group,  2010 


Caring 
is  better  than 

enforcing 


But  the  project  is  half 
over! 


x. 


You  must  follow  the 
process. 


But  the  project  is  half 
over! 


Okay,  good  point.  Let's 
think  about  what  would 
make  sense  here. 


"Complying  for 
compliance's  sake 
ravages  the 
operation  and 
firmly  entrenches 
self-defeating 
cycles  of  continued 
mediocrity." 


Hillel  Glazer, 

High  Performance  Operations 


Don't 

change  the  rules 

on  a  project  that's 
already  been 

planned 


unless  you're  willing  to  toss 


cost  schedule  quality 


A 


out  the... 


why? 


A  change  here... 


...applied  to 
the  project 

...will  impact  the 

project  plan 


4100 


Appraisals 


Hire  an  appraiser 

with  a 

business  value 

mindset 


maybe  not  him 


Don't 

under-estimate 
the  cost  of 

evidence  collection 


Average  total*  effort 

2500-3000 

hours 

for  small**  ML  3  appraisals 


*all  appraisal  activities,  including  but  not  **2  projects,  6  team  members  +  Lead, 

limited  to  evidence  collection  first  appraisal  at  ML  3,  CMMI  SE/SW  vi.i 


Source:  Value-Based  CMMI  AppraisalTechniques,  Systems  and  Software  Consortium  Inc.,  2006 


Total  appraisal  effort 


Source:  Value-Based  CMMI  AppraisalTechniques,  Systems  and  Software  Consortium,  2006 


SCAMPI 


vi. 3  may 
help  with 
this  a 
bit... 


Standard  CMMI F  Appraisal  Method  ior 
Process  Improvement  (SCAMPIbm)  A. 
Version  1.3:  Method  Ddiniikm  Document 


But  you 
still  need 
to  clearly 
identify... 


Process  development 

Ensure  that  your  process  developers  appreciate  the  model 
Involve  people  that  do  "real"  work 
Use  project  planning  to  plan  your  PI  project 
Develop  useful  tailoring  guidance 


Process  deployment 

Don't  just  announce  the  existence  of  a  book 
Caring  is  better  than  enforcing 

Don't  change  the  rules  on  a  project  that's  already  been  planned 


Appraisals 

Hire  an  appraiser  with  a  business  value  mindset 
Don't  under-estimate  the  cost  of  evidence  collection 


Super. 


But  whatever 
happened  to 

Pam? 


o  0O0 


r  ^ 

She  wakes  up  one  morning  and... 

L  1 


realizes  her  CMMI® 
experience  was  all 


so  she  grabs 

some 


and  heads 

off  to 


where 


CMMI 

for  Development 

Guidelines 


P 
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Integration 
and  Product 

Impr°venicnt 


Third  Edition 
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is  used,  but 


Compliance 


doesn't 


mi 


of  developing  processes  that 


improve 


o 


Appraisal 

preparation 


doesn't 


project 


progress. 


and  much  to 
Pam's  delight 


Pam's 

happy! 


and  all  I  ask 
of  you  is  to 


Please  stop  the 

mindless 

bureaucracy 

and  instead  use  the  CMMI®  to 

legitimately 

improve 

your  operations. 


•  The  2010  CMMI  Success  Factor  Survey.  Vienna  VA:  ACME  Process  Group, 
November  2010. 

•  Caldwell,  Laura,  Sam  Fogle,  and  Gene  Jorgensen.  Value-Based  CMMI  Appraisal 
Techniques.  Version  01.00.  Herndon  VA:  Systems  and  Software  Consortium,  2006. 

•  Chrissis,  Mary  Beth,  Mike  Konrad,  and  Sandy  Shrum.  CMMI  for  Development: 
Guidelines  for  Process  Integration  and  Product  Improvement.  3rd  ed.  Boston: 
Addison  Wesley,  2011. 

•  Glazer,  Hillel.  High  Performance  Operations:  Leverage  Compliance  to  Lower 
Costs ,  Increase  Profits ,  and  Gain  Competitive  Advantage.  ist  ed.  Upper  Saddle 
River  NJ:  FT  Press,  2012:31. 

•  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  A , 

Version  1.3:  Method  Description  Document.  Pittsburgh:  Software  Engineering 
Institute,  March  2011. 


Leading r  j 

—Ed lie 


k 


CONSULTANTS 


■==  SEI  Partner 


Bill  Smith  MarySegnit 

More  info? 

bill@cmmitraining.com 

www.CmmiTraining.com 


Rock'n  CMMI  Training 


...and  Appraisals! 


Lending  rk 

Edder 


SEI  Partner 


CO  IM  S  kJ  LTA IN  TS 


Questions 
from  the 
Trenches 

What  Over  1000 
Students  Want  to  Know 
Most  About  the  C M M I ® 


Bill  Smith  CEO 

Leading  Edge  Process  Consultants  LLC 
www.CmmiTraining.com 


This  presentation  is  being  delivered  at  the  11th  Annual  NDIA  CMMI 
Technology  Conference  and  User  Group  in  Denver,  Colorado,  USA,  on 
November  16,  2011.  All  slides  contained  herein  are  Copyright  2011  by 
Leading  Edge  Process  Consultants  LLC.  Basically,  you’re  not  allowed  to 
copy,  modify,  or  otherwise  use  any  of  them  without  our  written 
permission.  Please  respect  the  fact  that  I  left  our  logo  and  copyright 
information  off  the  individual  slides  to  make  them  look  a  bit  cleaner. 
(Cool,  huh?)  Plus,  the  Software  Engineering  Institute  (SEI)  would  like  you 
to  know  that  SCAMPI,  SCAMPI  Lead  Appraiser,  and  IDEAL  are  all 
service  marks  of  Carnegie  Mellon  University.  Wait,  you’re  still  reading  this? 
Good  for  you!  There’s  more.  CMMI  is  registered  in  the  US  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University.  Sorry,  but  my  lawyer 
made  me  add  all  this  stuff,  and  he  only  speaks  legalese.  I  promise  the 
rest  of  your  time  with  me  will  be  a  bit  more  interesting.  Thank  you  for  your 

patience.  Bill. 
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Who  am  I? 


Module  9 

Improvement 

Infrastructujf 


62  times,  to  1102  students 


1102 

> 


The  number 
of  hours 
Lindsay  Lohan 
has  spent 
in  court! 


In  class,  I've  been 

asked  ioo's  of 


Some  before 
class  even  starts 


Does  your  class 
have  a  dress  code? 


(Name  Withheld) 


Obviously 

not. 


(Name  Withheld) 


I'll  be  presenting 

a  sampling  of 

those  questions 

with  help  from... 


Vishal  Agarwal;  Project  Manager;  REI 
Systems  Inc. 

Bill  Altman;  Principal  Engineer;  SCRA/ATI 

Dale  Bingham;  Director  of  Quality 
Management;  PSI  Pax 

Susan  Carlson;  SVP  &  Chief  Operating  Officer; 
A+  Government  Solutions 

Matthew  Carney;  IT  Specialist;  U.S. 
Department  of  Education 

Judy  Engle,  PHR;  HR  Manager;  Tribalco,  LLC 
Gayle  Giblin;  Industry  Standards  Lead;  IBM 

Tim  Gollner;  Quality  Manager,  HES  Director; 
G&B  Solutions 

Ned  Gubbi;  Quality  Manager,  Sunhillo 
Corporation 

Sharon  Howington;  Technical  Writer;  SMS 
Data  Product  Group,  Inc. 

Sandra  Kinsey;  PI/QA  Engineer;  Primus  ASRC 


Erika  Kohnke;  StaffTech,  Inc. 

David  Kresheck;  Senior  CMMI  Specialist; 
CGI 

Andy  Lisko;  Mgr  IT  Processes,  Quality  & 
Compliance;  United  Illuminating 

Mike  Mezeul;  Sr.  Director,  Adva  Optical 
Networking,  Inc. 

Shannon  Quinn;  Quality  Management 
Specialist;  Delta  Solutions  and 
Technologies 

Felicia  Stuckey;  Principal  Strategist; 
Visions  Strategic  Marketing 

Will  Swann;  Director;  Technatomy 
Corporation 

Lindsey  Swanson;  Society  for  Human 
Resource  Management 

Bob  Swenson;  Director  of  Quality;  C-Far 
Services 


Vishal  Agarwal;  Project  Manager;  REI 
Systems 

Bill  Altman;  Principal  Engineer;  SCRA/ATI 

Dale  Bingham;  Director  of  Quality 
Management;  PSI  Pax 

Susan  Carlson;  SVP  &  Chief  Operating  Officer; 

A+  Government  Solutions 

Matthew  Carney;  IT  Specialist;  U.S. 
Department  of  Education 

Judy  Engle,  PHR;  HR  Manager;  Tribalco,  LLC 
Gayle  Giblin;  Industry  Standards  Lead;  IBM 

Tim  Gollner;  Quality  Manager,  HES  Director; 

G&B  Solutions 

Ned  Gubbi;  Quality  Manager,  Sunhillo 
Corporation 

Sharon  Howington;  Technical  Writer;  SMS 
Data  Product  Group,  Inc. 

Sandra  Kinsey;  PI/QA  Engineer;  Primus  ASRC 


Erika  Kohnke;  StaffTech,  Inc. 

David  Kresheck;  Senior  CMMI  Specialist; 

CGI 

Andy  Lisko;  Mgr  IT  Processes,  Quality  & 
Compliance;  United  Illuminating 

Mike  Mezeul;  Sr.  Director,  Adva  Optical 
Networking,  Inc. 

Shannon  Quinn;  Quality  Management 
Specialist;  Delta  Solutions  and 
Technologies 

Felicia  Stuckey;  Principal  Strategist; 
Visions  Strategic  Marketing 

Will  Swann;  Director;  Technatomy 
Corporation 

Lindsey  Swanson;  Society  for  Human 
Resource  Management 

Bob  Swenson;  Director  of  Quality;  C-Far 
Services 


who've  given 

me 


A  mild  disclaimer 


In  this  presentation,  I’ll 
associate  each  student  with  a 
question  that  IVe  actually  been 
asked  in  class.  In  most  cases, 

that  specific  student  did  not 
ask  that  specific  question... 


except  for 

Susan 


I  have  an  important 
meeting  on  Thursday. 
Is  it  okay  if  I  miss  part 
of  class? 


No  problem. 


HI  just  give  you  a 

make-up 

assignment 


I  have  a  brand  new  SUV 


VIRGIN  rA^j  I 


that  could  really  use  a  nice  hand-waxing 


Let's  get  started 


Why  do  you  think 

requirements  are  so 

hard  to  get  right? 


So  many  reasons! 
Here's  one:  the  English 

language... 


If  you  can  mess  up 

the  requirements  for 

what  to  write  on 
a  birthday  cake 

is  it  any  wonder 
we  can  have  problems  with 


...harderthings? 


I  wasn't  at  that 
meeting,  but... 


When  considering  the  PAs  that  are  at 

ML2  vs.  ML3 

don't  think 

What  comes  first  in  the  life  cycle. 

Think 

Level  of  risk  to  a  project. 


Scope  creep  is  a  major  risk. 


Can  you 
take  out  the 
trash,  hun? 


Sure,  no 
problem! 


The  next  morning. 


Did  you  take 
out  the 

recycling? 


No. 


Say  you're 
developing 

ruggedized 

equipment 

that  must... 


survive  high  temperatures 


and  sandstorms 


You  could 

just 


find  a  convenient  desert... 


sit  there... 

j 


and  wait... 


for  days...  weeks... 
months... 


for  your  target  temperature... 


And  when  that 

special  day  arrives 

and  it's 

finally  130  degrees 

and  your  equipment  works 
and  you're  ready  to  leave 

and  you 

showered  for  the  first  time  in  60  days# 

you  realize 

you  can't  go  home 

because... 


Or  you  could 

just 


stick  the  equipment 

in  an  oven... 


_  ...and  throw  sand  at  it 


(An  oversimplification,  of  course.  But  you  get  the  point.) 


The 

operational  environment 

may  not  provide  you  with  all  the 

environmental 

characteristics 

you  want  to  test,  in  a 

reasonable  timeframe 

at  a 

practical  cost. 


I'm  a  DJ.  One  of  my  buddies 
attended  your  class  and  said 
you  applied  DAR  to  selecting 
music.  Really? 


Glad  you  asked. 


The  situation: 

-  I  targeted  a  retro-style,  CD- 
playing  jukebox  for  my 
basement 

-  The  capacity  is  100  CDs 

-  50  CDs  will  be  dedicated  to 
classic  rock  artists  -  one  CD 
of  each  artist's  best  songs 

Which  50  artists  should  have  a 
CD  in  my  jukebox? 


''All  the  News  That  Fits  ~ 


rollin^sionf.coin 


I  searched  through  lists  for 
potential  artists: 

—  Rock  &  Roll  Hall  of  Fame 

inductees 

—  Rolling  Stone  Magazine 

•  “100  Greatest  Artists” 

•  Cover  of  1000th  issue 


Critical  Acclaim 


•  Rolling  Stone 
Magazine 
ranking 

•  Rock  &  Roll 
HOF? 

•  Cover  of  Rolling 
Stone  #1000? 


I  decided: 


which  criteria 
were  important 


how  important 

each  was 


how  I'd 
measure  each 


Popularity 


Billboard 

magazine 
singles  ranking 
RIAA*  album 
sales 


My  Taste 


dar 


I  decided  howto  combine  all  this  information. 


Critical  Acclaim 

Popularity 

My  Taste 

Total 
(0  to  1) 

Select 

Svalua^-?^ 

Methods 

Artist 

30% 

30% 

40% 

Rock  and 
Roll  Hall 
of  Fame? 
(Thru 
2006) 

Cover  of 
Rolling 
Stone 
Magazine 
#1000? 
(5/18/06) 

Rolling 

Stone 

Magazine 

Ranking 

(4/15/04, 

4/21/05) 

Points 
(0  to  1) 

Billboard 

Magazine 

Singles 

Ranking 

(Thru 

12/27/03) 

RIAA  Album 
Sales 
(Millions, 

Thru  7/31/06) 

Points 
(0  to  1) 

Numberof 
5-Star 
Songs (My 
Ratings,  20 
Max) 

Points 
(0  to  1) 

...including  the  formulas  that  l7d  use  to  derive  each  artist's  final  rating 


I  ended  up  with  a  sorted  list  of  over  200  artists. 


Evaluate 
Alternative  1 
Solutions  l 

Artist 

Critical  Acclaim 

Popularity 

My  Taste 

Total 
(0  to  1) 

30% 

30% 

40% 

Rock  and 
Roll  Hall 
of  Fame? 
(Thru 
2006) 

Cover  of 
Rolling 
Stone 
Magazine 
#1000? 
(5/18/06) 

Rolling 

Stone 

Magazine 

Ranking 

(4/15/04, 

4/21/05) 

Points 
(0  to  1) 

Billboard 

Magazine 

Singles 

Ranking 

(Thru 

12/27/03) 

RIAA  Album 
Sales 
(Millions, 

Thru  7/31/06) 

Points 
(0  to  1) 

Number  of 
5-Star 
Songs (My 
Ratings,  20 
Max) 

Points 
(0  to  1) 

The  Beatles 

• 

• 

1 

1.00 

2 

169.5 

1.00 

20 

1.00 

1.00 

Elvis  Presley 

• 

• 

3 

■97 

1 

118.5 

0.88 

20 

1.00 

0.95 

The  Rolling  Stones 

• 

• 

4 

■95 

9 

65.5 

0.69 

20 

1.00 

0.89 

Led  Zeppelin 

• 

• 

14 

.81 

109.5 

0.60 

20 

1.00 

0.82 

Bob  Dylan 

• 

• 

2 

.98 

36.0 

0.35 

20 

1.00 

0.80 

The  Beach  Boys 

• 

• 

12 

.83 

20 

21.5 

0.45 

20 

1.00 

0.78 

Stevie  Wonder 

• 

• 

15 

■79 

5 

19-5 

0.49 

20 

1.00 

0.78 

Michael  Jackson 

• 

• 

35 

■55 

6 

60.5 

0.68 

20 

1.00 

0.77 

Bruce  Springsteen 

• 

• 

23 

.69 

87 

62.5 

O.5I 

20 

1.00 

0.76 

Prince 

• 

• 

28 

.63 

21 

39-5 

0.54 

20 

1.00 

0.75 

Here  are  the  Top  10. 


# 

Artist 

Pts. 

# 

Artist 

Pts. 

# 

Artist 

Pts. 

1 

The  Beatles 

1.00 

18 

Eric  Clapton 

0.64 

35 

CCR 

0.51 

2 

Elvis  Presley 

0.95 

19 

Neil  Young 

0.64 

36 

Van  Halen 

0.50 

3 

The  Rolling  Stones 

0.89 

20 

Aerosmith 

0.64 

37 

Chuck  Berry 

0.49 

4 

Led  Zeppelin 

0.82 

21 

Rod  Stewart 

0.62 

38 

Bob  Marley 

0.49 

5 

Bob  Dylan 

0.80 

22 

Fleetwood  Mac 

0.60 

39 

Steely  Dan 

0.49 

6 

The  Beach  Boys 

0.78 

23 

The  Ramones 

0.60 

40 

Simon  &  Garfunkel  0.48 

7 

Stevie  Wonder 

0.78 

24 

Jimi  Hendrix 

0.59 

41 

Sam  Cooke 

0.47 

8 

Michael  Jackson 

0.77 

25 

The  Clash 

0.58 

42 

TheTemptations  0.46 

9 

Bruce  Springsteen 

0.76 

26 

Pink  Floyd 

0.58 

43 

Blondie 

0.46 

10 

Prince 

0.75 

27 

Marvin  Gaye 

0.57 

44 

Ray  Charles 

0.44 

11 

Elton  John 

0.74 

28 

The  Police 

0.57 

45 

AC/DC 

0.43 

12 

Madonna 

0.74 

29 

Bob  Seger 

0.56 

46 

The  Supremes 

_  0.43 

13 

U2 

0.73 

30 

David  Bowie 

0.55 

47 

Queen 

|  dar 

14 

The  Who 

0.66 

31 

Tom  Petty 

0.55 

48 

Eminem 

L  splfi 

■J 

15 

Billy  Joel 

0.66 

32 

The  Doors 

0.53 

49 

Boston  1 

s°/&  7 

i6 

The  Eagles 

0.66 

33 

John  Mellencamp 

0.52 

50 

Paul  Simon  / 

17 

Johnny  Cash 

0.66 

34 

Journey 

0.52 

L 

Artist 


change  request 

Originator 

CMMl  Student 

Si  M«*» '»•»<« 


Rationale 

[  Metallica  rocks. 


|  Disposition 

I 
K 
1 


Can  you  do  a  quick 
walkthrough  of  the 

engineering  PAs 

again? 


No  problem,  Tim. 


The  following  slides  use 

graphics 

I've  adapted  from 

"How  Projects  Really  Work" 

available  at 

www.projectcartoon.com 


really  needed 


What  the  customer 
really  needed 


RD 

SP  2.1 


How  the  project  leader 
understood  it 


Establish 
Product  and 
Product 
Component 
Requirement*; 


How  the  customer 
explained  it 


What  the  customer 
really  needed 


How  the  customer 
explained  it 


^isSST* 

Documentation 


How  the  customer 
explained  it 


IHwrEfre  programmer 


wrote  it 


How  the  project  was 
documented 


What  the  customer 
really  needed 


"Mwrffre  programmer 
wrote  it 


How  the  project 
was  documented 


Perform 


validation 


How  the  customer 
explained  it 


What  operations 
installed 

How  it  performed 
under  stress 


How  the  project 
leader  understood 


How  the  customer 
explained  it 


How  the  analyst 
designed  it 


Hmrflrc  programmer 
wrote  it 


What  operations 
installed 


What  the  customer 
really  needed 


How  the  project 
was  documented 


How  it  performed 
under  stress 


iSwing 


yant  to  share 
thls  Particular 
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You  said  institutionalization  is 

when  you  can  say  "that's  the 
way  we  do  things  around  here." 
Is  that  always  good,  though? 


Excellent  point,  Lindsey! 

Not  always... 


When  you 

institutionalize 

processes, 

they  become 

organizational 

habits. 


Habits  can  be  good... 


€ 


You  can 

use  the  CMMI®  to 

create  processes 

that  are 

repeatably... 


Just  say  no! 

to  bad  processes 


Check 

out 


My  2010  NDIACMMI®  Conference  Presentation 


A 


H 


_ 

3i(to*r 


10.  Don’t  Ignore  the  Nays| 
Be  Right 

You  May  Lose  People 
Okay 

“And  Then  a  Miracle  | 
a  Deployment  Plan 
Process  lmprovemen| 
Compliance  Are  Not  ! 
Keep  an  Open  Mind 


e  K^eb  su  obeu  yyiuq 
COUib|i9uce  y/ie  moj  ^ 

V  btocea*  lUJbioAeuj 


10th  Annual  National  Defense  iru 
CMMI®  Technology  Cpnftrence  - 

NovembenM,  2010  1  /* 

Denver,  Colorado,  USA 

Tmck  1 ;  GMMI  anti  Proems  Impmvgmani 
Session  11315,  aizsmOvek2:lS-3.Wpm 


istrfal  Association 
id  User  Group 


i  DOESN'T  KILL  YOU  MAKES  YOU  STRONGER: 

’ROCESS  IMPROVEMENT 
1  LESSONS  LEARMM^ 

mjapw 

Bill  Smith,  CEO 

Leading  Edge  Process  Consultants  LLC 

www.Cmm  iT raining .  com 


Or  one  of 
the  many 
books 
written 


CMMI 
Survival  Guide 


\ 


Just 

Enough 

Process 

Improvement 


Suzanne  Garcia 
Richard  Turner 

foreword  by  Mike 


...people 
much 
smarter 
than  me! 


Could  you  say  a 
little  more  about 
the  different  types 

of  appraisals? 


This  is  not  a  class  on 
appraisals,  and  I'm  not 
a  Lead  Appraiser.  But 
since  we  have  a  few 
extra  minutes... 


The  next  four  slides  are  based 
on  quotes  from  CMMISCAMPf 

^Willed:  Appraisals  f°r 
eK|8ss  Improvement 

al'Addison-Wesley,20°5 


SCAMPI 

Fun 


an  appraisal 
in  which  the 

objective 
evidence 
presented 
to  the  team 
is  very  thin 


All-You-Can-Eat  SCAMPI 


appraisal  in 
which  the  team 
ped  with 


is  swam 

objective  evidence  for 
every  model  subpractice 


-  Jr, 

an  appraisal  whose  lead  appraiser  comes 
from  within  the  company  being  appraised 


Sushi  SCAMPI 


an  appraisal  whose  results  seem  Fishy 


What  are  some 
characteristics  of 

maturity  level  1 

organization? 


I  thought  you'd  never  ask. 

Thanks,  Dale, 
for  setting  up... 


WWW 


You  think  a  su6practice 
is  a  test  run  for  an  underwater  boat. 


You  Know  You're  ML  i  When 


Establish 
Operational 
Concepts  and 
Scenarios 


You  believe  an 


Operational 


scenario 


involves  an  ill- 


fitting  gown  that 
4  ties  in  the  rear 


You  KnowYou're  ML  1  When: 


Your  integration 
procedures 

use  the  phrase 
"duct  tape" 
three  or  more  times 


Your  primary  ' 

causal  analysis 
tool  is  the 
Blame  Allocation 
Matrix. 


Select 

Evaluation 

Methods 


You  think  a 

maturity  level 

is  something 
you  attain 
when  you're 
old  enough  to 
join  the  AARP 


The  End 

(kinda) 


1 

Join  me  again  right  here  at  2:15! 

J 


l.rmlintf 


ml  mu  ^ 

Mfirr 


SEIParlner 


Why 

Project 

Managers 

(Understandably) 

Hate 

the  CMMI* 


Bill  Smith  CEO 

Leading  Edge  Process  Consultants  LLC 
www.CmmiTraming.com 


CMMI 

for  Development 


Guideline* 
for  Process 
Integration 
and  Product 
Improvement 


Tmi  *n  Edition 


Mary  Beth  Chrutn 
Mike  Konrad 
Wd>  Shrum 


MMM  CWUJ!119IUIU3  C0UJ 

rG9qiU0  Eq^6  btOC622COU2n|I9U(2  rrc 

Bill  2U*\t\)  CEO 


Leading r  j 

—Ed &e 


k 


CONSULTANTS 


-^p-  SEI  Partner 


Rock'n  CMMI  Training 


Bill  Smith 


...and  Appraisals! 


Mary  Segnit 


More  info? 

bill@cmmitraining.com 

www.CmmiTraining.com 


Leading r  j 

—Ed &e 


k 


CONSULTANTS 


-^p-  SEI  Partner 


Rock'n  CMMI  Training 


Bill  Smith 


...and  Appraisals! 


Mary  Segnit 


More  info? 

bill@cmmitraining.com 

www.CmmiTraining.com 


A  Tale  of  Two  Cultures 

NDIA  CMMI  Conference  -  November  2011 


Presenters: 


Marie  Johnson 


LOCKHEED  ll/l  A 


Beth  Layman 


LAYMAN  &  LAYMAN 


Agenda 

*  Scope 

*  Two  Cultures 

*  Architecture 

*  Appraisals 

*  Change  Management 

*  Plan  Forward 
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FDOC  Overview 


Mission  Support  Operations 
Contract 


J  Sustainment  and 
Operations 

£  Mission  Control 
Center 

£  Integrated  Planning 
System 

£  Houston  Support 
Room  -  Moscow 

£  Ground  Systems 
Development 
Environment 

£  Common  Development 
Environment 

J  Operations  Technology 
Facility  Support 


New 


■1  Constellation  Training  Facility 
(CxTF) 

and  PAS  for  Cx 


Space  Program  Operations 
Contract 


*  Shuttle  System  and  Cargo 
Engineering  and  Integration 

*  Flight  Element  Processing 

•  Orbiter 

•  Solid  Rocket  Booster 

•  Flight  Software 

•  External  Tank 

*  Launch  and  Landing  Operations 

*  Flight  Operations 

d  Mission  Control  Center  Automation 
System 

d  ISS  MOD  Avionics  Reconfiguration 
System 

d  Day  of  Launch  Initialization  Load 
Update 

^  Shuttle  Mission  Training  Facility 
^  User  Application  and  Supporting  Data 
Files 

^  Software  Production  Facility 


S  FDOC  Scope 
•  SPOC  /  IMOC 
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Two  Cultures  to  One 


r  ^ 

Lockheed 

Martin 


•  One  set  of  Established  Processes 

•  Large  development  projects 

•  HW  and  SW  Integration 

•  12M  Lines  of  code 


r  i 

United 

Space 

Alliance 


FDOC 

Goal 


•  Four  Processes 

•  Some  work  CMMI  Level  3  compliant 

•  Majority  of  the  work  less  than  3  month  effort 

•  60M  lines  of  code 


•  One  Process 

•  CMMI  Level  3  for  Development 


Path  to  Consolidation 


*  Conducted  a  series  of  LM21  events  for  consolidation 
of  processes  from  two  predecessor  contracts 

-  29  events  in  2009  and  7  events  in  2010 

-  Didn’t  get  to  a  steady  state,  continued  evolving 
process 

*  Considered  USA  Common  Software  Process  (CSP) 
but  found  it  too  limited  in  scope  (Small  SW  jobs 
only) 

-  Did  not  consider  their  lessons  learned 

*  Chose  a  Level  5  process  set  that  did  not  meet  our 
business  model 


Result  was  that  every  employee  had  to  learn 
something  new 
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Road  to  CMMI  for  Development 


*  Early  Successes 


-  Established  the  Process  Asset  Library 


Organizational 
Standard  Process 
(OSP) 


-  Established  the  Organizational  Process 
Group 


-  Use  of  myFDOC  and  collaboration  tools 


*  Struggles 


-  Created  new  processes  vs.  Document 
what  you  do 

-  Developed  action  plans  late 

-  Internal  audits  late 

-  Choice  for  not  using  CMMI  for  Services 
was  based  on  lack  of  experience 

•^Acknowledge  that  change  is  hard 


Appraisal’s 

*  Approach  included  the  LM  Continuous  Assessment 
Method  (CAM) 

-  Six  phases  to  continuously  assess  compliance 

*  SCAMPI  B  was  done  as  a  risk  mitigation  to  identify 
any  weaknesses  in  the  model  implementation 

-  Weaknesses  were  identified  and  the  SCAMPI  A  was 
delayed 

*  SCAMPI  A  successfully  conducted  after  all 
weaknesses  were  corrected  and  reviewed  by  the 
Lead  Appraiser 


pud’s 


Phase  I  (CAM) 


•  1  piece  of 
evidence  per 
project 

•  Engineers 
provided  data 


SCAMPI  B 


•  Training  from  the 
LA  (not  followed) 

•  New  projects 
selected 

•  Collected  data  by 
Process  Area 

•  Data  duplicated 


SCAMPI  A 


•  Compliant  with 
the  training 

•  Additional 
projects  selected 

•  Collected  data  by 
projects 

•  Small  team 


LA/Advisor  Going  In  Perspective 

•  People 

-  PI  team  -  too  focused  on  develop  vs.  implement 

-  Lack  of  change  leadership 

•  Processes 

-  High  Level/Generic/CMMI-speak 

-  Text-based 

-  Tailoring  vision  cloudy 

•  Tools 

-  Great  tooling;  but  disconnected  from  process 

•  Other 

-  Measures  focused  solely  on  contract  requirements 

-  “Project”  selection  proved  difficult 


Leadership  Scale 


Management  Positions 


Systems  Thinker 


Apathetic 
“There  Already” 


Not  my  job 


Change  Adverse 
Passive 


Narrow  View 


Engaged 


Supporter 

- > 

Let’s  fix  it! 


Degree  of  Engagement 
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Change  Management 


What  We  Did 


•  Drafted  a  Plan 

•  Established  a 
Baseline 

•  Selected  Framework 

•  Identified  Resources 

•  Designed  Solutions 

•  Monitored  Progress 


What  We  Should  Have 
Done 


•  Share  a  Vision 

•  Communicate 
Expectations 

•  Be  Honest  About 
Changes 

•  Handle  Resistance 

•  Reward  Correct 
Behaviors 
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Communication 


•  Vision 

*  Roles  and 
Responsibilities 


Porcelain 


Press 


What  is  CMMI? 

•  The  acronym  stands  for:  Capability  Maturity  Model  Integration 

•  In  a  nutshell,  this  is  what  it  means:  CMMI  is  a  set  of  process  models  that 
compare  an  organization's  process  capability  (maturity)  against  industry 
standards. 

What  is  SCAMPI? 


•  The  acronym  stands  for:  Standard  CMMI  Appraisal  Method  for  Process  Improvement 

•  What  it  is:  Organizational  appraisal  using  a  CMMI  model.  Three  SCAMPI  methods  of  increasing  level  of  rigor  are 
defined  (SCAMPI  A,  B,  and  C).  Only  the  most  rigorous  method  (SCAMPI  A)  can  result  in  a  rating. 


FDOC’s  goal  is  CMMI  Maturity  Level  3  (Defined) 


Let's  Talk  FDOC  Organization  Standard  Process  -  CMMI  Level  3 

In  July  2010,  FDOC  deployed  its  Organization  Standard  Process 
(OSP).  In  December,  2010  FDOC  completed  a  CMMI  SCAMPI  B 
Appraisal  on  its  OSP.  Although  FDOC  has  made  huge 
improvements  on  institutionalization  of  the  OSP  some  weaknesses 
were  found.  Listed  below  are  some  global  weaknesses  that  all 
FDOC  employees  should  be  working  to  eliminate. 


SCAMPI  B  -Global  Weaknesses  Awareness 

•  Newly  deployed  processes  -  lack  of  adoption  throughout  the 
organization 

•  Audits  were  done  prior  to  process  awareness  and  no  follow-up 
of  findings 

•  Distributed  project  management  and  control  of  engineering  led 
to  confusion  of  roles  and  responsibilities 

•  Lack  of  project  measures  to  all  levels  of  work 

•  Practices  not  met:  Analysis  of  peer  review  data  and  use  of  historical  data  for  costing 

•  PIID's  were  disorganized  with  poor  Configuration  Management 


Why  is  CMMI  important  to  me? 

•  Not  only  does  CMMI  provide  a  proven  framework  for  program  planning  and  execution,  but  it  helps  each  of  us 
become  more  competitive  as  a  unit.  This  is  critical  in  today's  era  of  economic  struggles. 

•  As  a  systems  provider,  we  are  required  to  be  CMMI  Level  III  (minimum)  just  to  continue  to  have  the  opportunity 
to  keep  doing  what  we're  doing. 

•  Because  CMMI  methodology  is  performance-focused  with  measurable  impacts  on  business  effectiveness,  this 

provides  our  customer  with  concrete  evidence  of  our  benefit  to  them. 

•  When  CMMI  practices  are  used  properly,  it  facilitates  teaming,  process  integration,  and  benchmarking  which 
helps  all  of  us  to  be  more  effective  in  our  individual  roles. 

•  CM  M I  hel  ps  us  to  recognize  our  process  gaps  and  provides  ways  for  correcting  those  gaps  long  before  they 
become  an  issue  for  our  customer. 


Look  forward  to  further  CMMI  communication  that  can  lead  to 
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Handling  Resistance 


Meetings  with  Managers 
to  be  champions 


Lunch  with  Employees 
to  understand  concerns 
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Reward  Correct  Behaviors 


Play  CM  MI  Millionaire  ! 


Play  CMMI  Crossword  PuzzJel 


IRESERVEO 


FDOC 

CMMI 

Contest 

Winner 


11/21/2011 


Plan  Forward 


*  Consolidation  of  ISO  and  CMMI  Audit  checklist 

-  Audit  by  role  instead  of  by  process  area 

*  Creation  of  a  measurement  working  group 

-  Ensure  the  continuation  of  measurement  collection 
and  review 

-  Ensure  the  measures  are  value  add 

*  Promote  the  use  of  best  practices  across  FDOC 

*  Consolidation  of  Configuration  Management 
systems 

-  Eliminate  redundant  data 

*  Maintain  the  Organizational  Process  Assets 


LA/Advisor  Post  Mortem  Perspective 


*  CAM  and  LA  not  coordinated,  aligned 

-  audit/test  vs.  intent 

-  breadth  and  depth  of  practice  implementation 

*  FDOC  Process  Lead’s  tenacity  essential 

*  Changes  made  between  SCAMPI  B  and  A  brought 
processes  closer  to  actual  practice 

*  Ongoing  activity  post  SCAMPI  A  is  sure  sign  of  true 
maturity 

*  CMMI  for  Services  probably  a  better  fit 


16 


Customer 


CMMI  DEV  or  SVCs? 


System  Delivery  Schedules 

■  ■ 


MySRM 


c 


Only  a  small 
percentage  of  SRs 
Were  grouped  and 
managed  like  a  “project” 


□ 


Contract 

SLAs 


- \ 

MCCS 


- \ 

xxxx 


_ /  V _ / 
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Questions? 


Thank  You! 


□NREL 

NATIONAL  RENEWABLE  ENERGY  LABORATORY 

Creating  an  SQA  Program  at  NREL 

A  DOE  FFRDC  Program  based  on  the  NDIA/DoD  Sponsored  CMMI 


CMMI  Conference  2011 
Tim  Kasse 
16  November  2011 


NREL  is  a  national  laboratory  of  the  U.S.  Department  of  Energy,  Office  of  Energy  Efficiency  and  Renewable  Energy,  operated  by  the  Alliance  for  Sustainable  Energy,  LLC. 


NREL 


The  National  Renewable  Energy  Laboratory  (NREL) , 
located  in  Golden,  Colorado  is  the  only  Department  of 
Energy  (DOE)  laboratory  in  the  United  States  that  is 
“solely”  focused  on  renewable  energy  with  focus  on 
such  research  areas  as: 

-  Solar 

-  Wind 

-  Bio-Mass 

-  Hydrogen 


National  Renewable  Enerav  Laboratory  (NREL) 


Version  2.2 


SQA  Program  -2 


NREL -  2 


Research  Service  Facility  (RSF)  buildings  used  for 
staffing  are  the  most  efficient  in  the  world. 

-  Visitors  from  all  over  the  world  come  to  see  and  hear 
about  how  these  energy  efficient  buildings  were  built 
and  how  they  work 

-  Newest  RSF  building  scheduled  for  occupancy 
starting  in  December  will  be  zero-net  energy. 


National  Renewable  Enerav  Laboratorv  (NREL) 


Version  2.2 


SQA  Program  -3 


NREL -  3 


The  past  30  years  has  seen  NREL  evolve  as  a 
renewable  energy  facility,  non-profit,  government- 
funded  entity 

Today,  after  doubling  in  size  in  the  past  3  years,  NREL 
finds  itself  being  required  to  be  more  strictly  focused 
on  business  with  the  ability  to  prove  it  is  focused  on 
continuous  process  improvement  to  satisfy  both  the 
DOE  and  its  industry  and  academic  partners 

In  addition,  the  importance  of  Software,  together  with 
Systems  Engineering  has  risen  to  a  visibility  far 
beyond  that  of  its  past  30  years 

-  Along  with  that  visibility  goes  software  quality  and  the 
processes  that  product  it 


National  Renewable  Enerav  Laboratory  (NREL) 


Version  2.2 


SQA  Program  -4 


Software  Quality  Assurance 
versus  Process  Improvement 


In  the  CMMI-based  process  improvement  world, 
Process  Improvement  might  be  considered  the 
overarching  umbrella  with  Quality  Assurance  operating 
beneath  it. 

In  the  NREL-DOE  world  Quality  Assurance  /  Software 
Quality  Assurance  serves  as  the  overarching  umbrella 
with  process  improvement  operating  beneath  it. 


National  Renewable  Enerav  Laboratorv  (NREU 


Version  2.2 


SQA  Program  -  5 


SQA  Program  Approach 


CMMI  Continuous  Representation 
Incremental  process  improvement 
Problem  Solving  Approach 

-  Integrated  Models/Standards 

-  Lack  of  emphasis  on  the  specific  models,  standards  or 
techniques  such  as  CMMI,  Lean,  Six  Sigma,  ITIL,  PMI, 
Agile,  etc. 

“Slices  of  Process  Areas  for  Improvement” 

Just  in  Time  Training 
Hand-Holding  Coaching 
Progress  Checks 
Basic  Measurement 


National  Renewable  Enerav  Laboratorv  (NREU 


Version  2.2 


SQA  Program  -6 


SQA  Program  Approach  -  2 


Identifying  business  objectives  for  projects,  programs  and 
departments 

Providing  definitions,  descriptions,  papers,  references 

Using  Peter  Block’s  Consulting  Modes  (Expert, 
Collaborative  and  Observer) 

Borrowing  Peer  Review  concept  of  “Kin  Documents” 


National  Renewable  Enerav  Laboratorv  (NREU 


Version  2.2 


SQA  Program  -7 


Integrated  Approach  to 


Action  Planning 
(Incremental  Approach) 


Handholding  Coaching  Support  for 
Methods  and  Tools 


National  Renewable  Enerav  Laboratory  (NREU 


Version  2.2 


SQA  Program  -8 


Software  Quality 
Management 


Quality  Control  and 
Quality  Assurance 


Quality  Control  evaluates  the  products 

-  Checks  the  quality  of  the  product(s) 

•  Is  the  product  within  tolerance? 

•  Is  the  product  (or  life-cycle  work  product)  of  an 
acceptable  quality? 

-  Tools  and  Techniques 

•  Reviews 

•  Tests 

•  Inspections 

•  Simulations 

•  Observations 

•  Demonstrations 


National  Renewable  Enerav  Laboratorv  (NREL) 


Version  2.2 


SQA  Program  -10 


Quality  Control  and 
Quality  Assurance  -  2 

Quality  Assurance  evaluates  the  process 

-  Checks  that  the  process  is  working 

•  Is  the  process  being  followed? 

•  Are  the  QC  checks  being  applied? 

•  Are  the  QC  checks  efficient? 

•  Is  the  process  causing  quality  problems? 

•  Is  the  process  working  for  the  organization? 

-  Tools  and  Techniques 

•  Audits  /  Objective  Evaluations 

•  Assessments  /  Appraisals  /  GAP  Analysis 
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Software  Quality 
Management  Functions 

Software  Quality  Management  Functions  include: 

-  Setting  Quality  Goals  that  support  business  objectives 

-  Establishing  and  enforcing  a  Quality  Policy 

-  Planning  for  Quality  (Organizational  and  Project  level) 

-  Developing  Processes  (Organizational  and  Project  level) 

-  Establishing  the  use  of  Standards  and  Procedures 

-  Performing  multiple  levels  of  Testing 

-  Conducting  Peer  Reviews  throughout  the  product  lifecycle 

-  Designing  in  Quality  Factors  (i.e.,  safety,  security,  maintainability, 
reliability) 

-  Conducting  Quality  Audits  with  respect  to  product  quality 

-  Conducting  Quality  Audits  with  respect  to  process  quality 

-  Providing  visibility  into  the  process  and  product  quality  for  management 
through  Quality  Reporting 

-  Getting  non-compliance  issues  resolved  before  the  product  is  delivered 
to  the  customer  (looking  for  trends) 
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Software  Quality 
Management  Functions  -  2 

Software  Quality  Management  Functions  also  include: 

-  Implementing  complementary  Configuration  Management 
functions 

-  Identifying  Measurements  that  support  the  information  needs  of 
the  project  and  organization 

-  Conducting  Performance  Evaluations  to  ensure  the  system 
converges  to  established  performance  constraints 

-  Conducting  appropriate  Verification  functions 

-  Conducting  appropriate  Validation  functions 
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NREL  Software  Quality  Assurance  Program 


Developed  the  Software  Quality  Assurance  Program 
based  on  and  linked  to  the  existing  Quality  Assurance 
Program. 

-  Software  Quality  Assurance  Program  Architecture 
developed  and  placed  on  Share  Point 
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Software  Quality  Assurance  Program  -  2 


Emphasis  of  the  SQA  Program  Architecture  is  broken  into 
four  major  areas: 

-  Business  Focus  -  (completely  populated) 

-  Technical  Focus 

•  Software  Process  Asset  Library  defined  to  more  detailed  levels 

•  Software  Lifecycle  Descriptions  defined  (Waterfall,  V-Model,  Waterfall) 

•  Software  Design  Methods  defined 

•  Process  Descriptions  started  and  available  for  Requirements, 
Inspections  and  Risk  Management 

-  Support  Functions  Focus  -  Executive  Overview  for 
Configuration  Management,  Risk  Management,  Supplier 
Management,  and  Quality  Assurance 

-  Institutional  Focus  -  Topics  describe  the  QMS&A  approach  for 
technology  transition  of  knowledge  to  Projects,  software  systems 
training  that  is  available  to  support  projects  and  Hand-holding 
coaching  QMS&A  is  starting  to  provide 
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SW  Process  Asset  Library 


Process  Descriptions 

Lifecycle  Descriptions 

•Process  Architecture 

(With  Examples) 

•Sets  of  Standard 

•Waterfall 

Processes 

•Overlapping  Waterfall 

•Policies 

•V-Model 

•Procedures 

•Spiral 

•Standards 

•Incremental 

•Guidelines 

•Evolutionary 

•Templates 

•Acquisition 

•Checklists 

•Agile 

•Starter  Kits 

Methods 

Tools 

•Object  Oriented 

•Project  Mgmt 

•Agile 

•CM 

•SCRUM 

•QA 

•SASD 

•Risk  Mgmt 

•Top-Down 

•Mind  Maps 

•Bottom-Up 

•Testing 

Good  &  Best  Practices 

•Lessons  Learned 
•Good  Examples 


Tailoring  Guidelines  / 

Measurement  Repository 

Graded  Approach  / 

•Actual  Project  Data 

Scaling 

•Measurement  Examples 

•External  and  Internal  to  the  Process  Descriptions 

•Guidance  for  Measures 

"Kin"  Documents 

Work  Environment 

Process  Improvement  Approaches 

•IEEE  Standards 

Standards 

•Big  Bang 

•CMMI 

•Development  Environments 

•Incremental  based  on  Project  Need 

•EIA-632 

•Testing  Environments 

•Incremental  based  on  Business 

•ITIL 

•Production  Environments 

Objectives 

•ISO  Standards 

•Web  Applications 

•"Constagedeous" 
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"Kin”  Documents 


IEEE  Software  Standards 

•730  -  Std  for  SQA  Plans 
•730.1  -  Guide  for  SQA  Planning 
•828  -  Std  for  SW  Configuration 
Management  (SCM)  Plans 
•1008  -  Std  for  Unit  Testing 
•1012  -  Std  for  SW  VER  and  VAL 
•1028  -  Std  for  Software  Reviews 
•1042  -  Guide  to  SCM 
•1058  -  Std  for  SW  Proj  Mgmt  Plans 
•1059  -  Guide  for  SW  VER  &  VAL  Plans 
•1219  -  Std  for  SW  Maintenance 
•1540  -  SW  Lifecycle  Processes  -  Risk 
Management 


INCOSE 

•EIA-731SE  CMM 
•EIA  -  632  Processes  for 
Engineering  a  System 


DoD  Standards 

•DoD-Std-2167a 

•DoD-Std-2168b 


ITIL  v3.0 

•Executive  Overview 
•Service  Strategy 
•Service  Design 
•Service  Transition 
•Service  Operation 

Lifecycles 

•ISO/EIA  12207  SW  Lifecycle  Processes 
•ISO/EIA  15288  SE  Lifecycle  Processes 


Quality 

•ISO  9001:2008  -  Quality  Management 
Systems  -  Requirements 
•ISO  9004:2009  -  Managing  for  the 
Sustained  Success  of  an  Organization 

-  A  Quality  Management  Approach 

•ISO/IEC  -  9126  -  Software  Engineering  -  Product 
Quality 

-  Quality  Model 

-  External  Metrics 

-  Internal  Metrics 

-  Quality  in  Use  Metrics 


Models 

•CMMI-DEV  vl.3 
•CMMI-SVC  vl.3 
•CMMI-ACQ  vl.3 
•ISO  15504  (SPICE) 
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"Kin”  Documents  -  2 


ISO  Standards 

•AS  9100  -  ISO  9000  for  Aerospace 
•TL  9000  -  Telecommunications 
•ISO  16949 -Automotive  Engineering 


Methods  and  Techniques 

•Six  Sigma 
•PMI  PMBOKv3 

•OPM  3  -  Proj  Mgmt  for  the  Organization 
•Lean 
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Training 


Classes  /  Workshops/  Tutorials  /  Presentations 

•EXAMPLES: 

•  Software  Quality  Engineering 

•  Software  Configuration  Management 

•  Requirements  Engineering 

•  Peer  Reviews  (Inspections) 

•  CMMI-DEV  vl  .3  -  SEI  Introduction  to  CMMI-DEV  vl  .3 
with  Kl  Additional  Value 

•  CMMI-ACQ  vl  .3  -  Kasse  Initiatives 

•  CMMI-SVC  vl  .3  -  Kasse  Initiatives 

•  CMMI-based  Assessments  (SCAMPI) 
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Incremental  Approach 


QMS&A  guides  its  NREL  customers  in  developing  an 
incremental  approach  to  the  process  improvement 
needs  it  has  identified  and  prioritized,  being  ever 
mindful  of  satisfying  the  business  objectives  as  well  as 
guidance  provided  by  the  CMMI  and  other  standards 
and  models 

Increments  of  approximately  3  months  in  duration  are 
designed  based  on  the  organization’s  business 
objectives. 

-  These  increments  are  populated  with  improvement 
tasks  that  represent  achievement  of  some  aspect  of 
planning,  developing,  testing,  controlling,  and 
managing  software  development 


National  Renewable  Enerav  Laboratorv  (NREL) 


Version  2.2 


SQA  Program  -21 


Hand-Holding  Coaching 

“Hand-holding”  support  is  “roll-up-your-sleeves”  support 
provided  to  help  implement  critical  processes  or  “slices”  of 
processes  on  projects.  Projects  can  expect: 

-  Knowledge  about  the  software  process 

-  Assistance  in  developing  process  descriptions 

-  Assistance  in  creating  a  Project  Quality  Plan 

-  Assistance  in  choosing  the  right  standards  for  the  project’s  needs 

-  Assistance  in  applying  the  tailoring  guidelines  to  the  organization’s 
process  descriptions  for  practical  use  by  the  projects 

-  Performing  quality  and  configuration  audits 

-  Assistance  in  setting  up  Peer  Reviews 
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Hand-Holding  Coaching  -  2 

Currently  being  focused  on: 

-  Requirements  Development  and  Requirements 
Management 

-  Risk  Management 

-  Software  Inspections 

-  Software  Safety 

-  Web  Applications  Development 

-  Change  Control  Boards 
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Current  Focus  Areas 


Configuration  Management 

(Serving  Project  Management  and  Quality) 

Identification 
Baselining 
Change  Control 

-  Change  Advisory  Boards 
Status  Accounting 
Configuration  Auditing 

-  Functional  Configuration  Audits 

-  Physical  Configuration  Audits 


National  Renewable  Enerav  Laboratory  (NREU 


Version  2.2 


SQA  Program  -25 


Hierarchies  of  CABs 


Business  Critical 


High  Risl 


Low  or  Medium  Risk 


Organizational  or 
System  Level  CAB 


Alliance  Leadership  Team 


V 


M 


Center  /Office 
Director 
Level  CAB 

i 


r 


Low  or  Medium  Risk 


S/W  Project  Level  CAB 


V 


Program  Level 
Level  CAB 


V 


A 


■ 


S/W  Project  Level  CAB 


Low  Risk 


V 


S/W  Project  Level  CAB 


National  Renewable  Enerav  Laboratory  (NREU 


Version  2.2 


SQA  Program  -26 


Web  Applications  Development  Process 


The  Web  Applications  Development  Process  is  serving  as 
the  platform  for  improvement  of  the  software  applications 
development  processes 

-  These  improvements  will  assist  the  development  of  many 
classes  of  software  and  are  some  of  the  first  steps  in  the 
development  of  the  overall  Software  Quality  Assurance 
(SQA)  program 

Draft  Policy  for  the  Web  Applications  Development  Process 
has  been  developed 

Web  Applications  Development  Process  has  been  drafted 
and  reviewed  by  the  Task  Force 

-  Has  been  submitted  to  the  NREL  review  process  to  be 
placed  into  the  NREL  Standard  Policies  and  Procedures 
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CMMI  Overview 

Process  Arees 

Optimising 

Focus  is  on  quantitative 
continuous  process 
improvement 

Causal  Analysis  and  Resolution 

Organizational  Performance  Management 

Quantitatively 

Managed 

Process  is  measured 
and  controlled 

Quantitative  Project  Management 

Organizational  Process  Performance 

Defined 

Process  is  characterized 
for  the  organization  and 
is  proactive 

Requirements  Development  Integrated  Project  Management 

Technical  Solution  Risk  Management 

Product  Integration  Decision  &  Resolution 

Verification 

Validation 

Mahgtjed 

Process  is  characterized 
for  projects  and  is  often 
reactive 

Requirements  Management  Configuration  Management 

Project  Planning  Measurement  and  Analysis 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Product  and  Process  Quality  Assurance 

Initial 

Process  is  unpredictable, 
poorly  controlled,  and 
reactive 

i 

CMMI-DEV  vl.3  Training 


Completed  CMMI-DEV  v  1 .3  4-Day  Training  Class  for  Enterprise 
Solutions  and  other  specialists  such  as  Internal  Audit. 

-  The  Capability  Maturity  Model  Integration  (CMMI)  was 
developed  by  the  Software  Engineering  Institute  (an  FFRDC)  to 
focus  on  the  Project  Lifecycle  from  requirements  elicitation  to 
transfer  to  the  Customers  /  End  Users  operational  environment. 

-  The  CMMI  model  is  referenced  in  DOE  O  414.1  D  as  one  of  the 
international  consensus  models  that  may  be  used  to  enforce 
414.1  D  requirements. 

-  The  CMMI  describes  requirements  for  development  activities 
that  not  only  support  the  development  lifecycle  but  dovetails  into 
the  Service  Management  activities  described  by  the  IT 
Infrastructure  Library  (ITIL  v3)  currently  being  taught  and 
implemented  in  ISO. 

-  Other  classes  are  being  scheduled  for  1st  quarter  of  FY12  and 
beyond. 
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Requirements  Engineering  Training 


Conducted  two  (2)  Requirements  Engineering 
Workshop  for  Enterprise  Solutions  with  HR,  and 
Business  Systems  Governance  representation. 

-  The  course  was  well  received  with  one  person  stating 
that  it  was  the  best  course  she  had  received  in  20 
years  on  the  topic 

-  Templates  for  developing  Software  Requirements 
Specification  and  Interface  Requirements 
Specification  have  been  developed  and  are  being 
used 

-  Checklists  supporting  the  SRS  has  been  built  and  put 
in  use 

-  One  more  RE  Workshops  is  scheduled  at  the  end  of 
1st  Q  FY12. 
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SQA  Program  Coaching 


SQA  Program  coaching  is  starting  to  take  place  to 
support  the  institutionalization  of  the  concepts  being 
offered  in  the  software  quality  training  classes 

-  Requirements  Engineering  coaching  for  Enterprise 
Solutions 

-  Risk  Management  Coaching 

-  Software  Safety  Coaching 
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Software  Safety 


Developed  Software  Quality  Assurance  (QA)  requirements 
specific  to  the  development  of  safety-related  software 

Collected  Software  Safety  research  material  from  the  Internet, 
the  Software  Engineering  Institute,  selected  government 
standards  on  Software  Safety  and  manuscripts  written  by  Dr. 
Nancy  Levinson  of  MIT. 

Started  the  development  of  a  Software  Safety  Guidebook  based 
on  research  material  and  NREL  software  safety  needs. 

Developed  a  Software  Safety  Presentation  based  on  the  current 
state  of  the  Software  Safety  Guidebook,  software  safety  research 
material,  CMMI,  and  the  SQA  program  that  satisfied  a  DOE 
compliance  action  item. 
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Software  Safety  -  2 


Applied  Software  Safety  concepts  by  working  with  the 
Process  Automation  software  vendor  developing  the 
software  for  the  SCADA  system,  part  of  Energy  Systems 
Integrated  Facility  to  ensure  their  software  safety  activities 
match  the  needs  of  any  software  safety  critical 
components. 

FY12  will  see  the  evolution  of  the  Software  Safety 
Guidebook  into  a  fully  developed  guide  for  all  of  NREL 
projects  that  involve  safety  and  particularly  software  safety. 

All  Software  Safety  materials  are  available  from  the  SQA 
Program  Share  Point  site! 
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Summary 


The  Software  Quality  Assurance  Program’s 
foundation  has  been  established,  based  on  the 
concepts  of  the  CMMI  and  its  implementation 
along  with  continuous  improvement  is  underway 
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Tim  Kasse  -  Contact  Information 


Tim  Kasse 

QMS&A 

Program  Manager  -  Software  Quality  Assurance  Program 
National  Renewable  Energy  Laboratory  (NREL) 

1617  Cole  Blvd. 

Golden,  Colorado  80401 
(303)  275  -  3285 
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Topic  Overview,  Objectives 
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Missile  Systems 


■  Raytheon  Missile  Systems  Organizational  Unit  overview 

■  About  the  vl  .3  Method  Definition  Document  (MDD)  scoping 
and  sampling  provisions 

■  Case  study:  RMS’  experience  with  applying  scoping  and 
sampling 

■  Experiences  and  Lessons 

■  Questions 

■  Biography 
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Raytheon  Missile  Systems  CMMI®  Background 


Raytheon 

Missile  Systems 


■  Raytheon  Missile  Systems 

-  One  business  of  Raytheon 

•  Characterized  by  high-volume,  complex  systems--“Rocket  Science” 

•  A  large  organization 

-  Appraised  at  Maturity  Level  5  for  CMMI®  for  Development  version  1 .2 
+IPPD  in  2009 

-  Next  appraisal  in  2012 

•  Selected  CMMI-DEV  vl.3  using  vl.3  of  the  Method  Definition  Document 
(MDD) 

•  To  use  MDD  vl  .3,  we  must  satisfy  the  scoping  and  sampling  provisions 
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Summary  of  vl  .3  Sampling  and  Scoping 


Raytheon 

Missile  Systems 


■  MDD  version  1 .3  calls  out  scoping  and  sampling  in 
section  1.1.4  (Implementation  Guidance) 

■  How  does  this  guidance  affect  my  organization?  (How 
bad  is  it?) 

-  Sources  of  information 

•  Software  Engineering  Institute  (SEI)  webinars 

•  SEI  staff  and  others  at  conferences 

•  Lots  of  reading  of  the  MDD  and  its  appendices 

•  Try  things 

•  Ask  questions! 
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Applying  MDD  vl.3  Sampling  Factors 


Raytheon 

Missile  Systems 


Sampled  projects  /  basic  units  (always  were)  representative  of  the  Organizational 
Unit  (OU) 


■  Define  and  evaluate  dimensions  which  cause  behavior  within  OU  to  vary 

-  Set  of  required  (to  be  considered)  sampling  factors  is  described  in  MDD  Appendix  F 

-  Select  those  sampling  factors  which  make  a  difference  in  process  execution  in  your  OU 

•  Use  sampling  factors  to  determine  your  subgroups 

-  Determine  whether  support  functions  exist  in  your  organization 

-  Use  selected  sampling  factors,  subgroups  and  support  functions  to  construct  a 
representative  sample  of  your  OU 

•  Document  these  in  your  appraisal  plan  and  package 

•  Might  result  in  more,  or  fewer,  instantiations  than  appraisals  conducted  under  MDD  vl  .2 


Note:  Terms  in  italics  can  be  found  in  MDD  glossary 

For  instance,  “Basic  Unit,”  “Sampled  Basic  Unit  (SBU),”  “Support  function,”  “[Sampled]  Subgroup” 
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Tips  for  Applying  the  vl.3  Sampling  and 
Scoping  Provisions 


Raytheon 

Missile  Systems 


■  Become  familiar  with  the  case  studies  in  MDD  Appendix  F 

■  Gather  information  about  your  OU 

■  Construct  various  scenarios  related  to  your  own  organization 

-  Remember  to  base  appraisal  objectives  on  your  goals 

-  Peer  review 

-  Have  others  ask  questions 

-  Iterate  as  necessary 

■  Vary  parameters  and  OU  dimensions  to  see  how  factors  affect  your  appraisal 

-  Constraining  the  OU  is  a  useful  tool 

.  You  may  find  that  two  appraisals  is  a  better  solution  than  one  big  appraisal 

■  Declare  and  use  appropriate  support  functions  to  reduce  instantiations 

-  Effort  is  required  to  initially  adjudge  whether  appropriate  to  declare  a  support  function 

-  Probable  effort  savings  over  the  appraisal  life  cycle 

■  Don’t  give  up! 
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Once  Approach  is  Decided,  Share  with 
Stakeholders 


Raytheon 
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■  Share  approach  with  lead  appraiser 

-  Process  will  be  iterative 

-  Don’t  assume  your  lead  appraiser  has  already  applied  all  the  provisions  of  sampling 
and  scoping  at  other  sites 

•  Your  OU  is  (probably)  unique 


■  Share  with  appraisal  team 

-  Also  iterative  process 

-  Start  with  internal  team  members 

-  Use  internal  members  to  educate  external  team  members 

-  Involve  all  members  of  team 

•  Including  any  who  don’t  often  lead 

•  Ensure  understanding  and  buy-in 

■  Share  with  other  stakeholders 

-  Appraisal  team  members  can  brief  sponsors,  managers,  basic  unit  leaders 
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Sampling  Factors  to  be  Considered 
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■  The  MDD  requires  organizations  to  construct  a 
representative  sample  of  the  organization.  Sampling  factors 
serve  to  identify  meaningful  differences  in  the  conditions 
under  which  work  is  performed  in  the  organization.  The 
following  sampling  factors  must  be  considered: 

-  Location 

-  Customer 

-  Size 

-  Organizational  Structure 

-  Type  of  work 

-  [Others] 
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Case  Study:  RMS  Application  of  SEI 
Sampling  Factors 
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RMS  appraisal  team  considered  the  potential  influential  contributors  to 
variation  in  the  conditions  under  which  our  work  is  performed. 


Sampling  Factor 

Applicability  to  RMS 

Location 

Single  location,  scope  of  appraisal  is  defined  as 
“RMS,  Tucson,  Arizona.” 

Customer 

Product  lines  account  for  different  customers. 

Size 

Period  of  performance  accounts  for  size. 

Organizational  structure 

Product  lines  account  for  different  organizational 
structure. 

Type  of  work 

Tiers  account  for  the  type  of  work. 
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Case  Study:  RMS  Factor  Analysis 
Summary 


Raytheon 

Missile  Systems 


■  All  criteria  which  are  within  scope  are  subsumed  by  RMS 
program  tiers 

-  RMS  has  defined  “tiers”  as  categories  of  like  programs  based  on  their 
end  use 

■  The  RMS  appraisal  team  mapped  sampling  factors  and 
selected  tiers  as  the  critical  factor  used  to  attain  the 
representative  sample  for  RMS 
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Observations  and  Experiences 


Raytheon 

Missile  Systems 


■  Become  familiar  with  the  intent  and  details  of  the  sampling  algorithm 

-  Examples  and  case  studies  helped  us  more  than  explanations 

•  Some  of  our  most  applicable  examples  came  from  Services  rather  than  Development 

-  Always  go  back  to  the  MDD  explanations  to  understand  what  flexibility  exists 


■  Version  1 .3  will  probably  drive  a  change  in  your  appraised  set  of 
instantiations 

-  More  diverse  work  and  project  /  basic  unit  types  can  anticipate  a  more  complex 
appraisal,  or  more  separate  appraisals 

-  More  consistent,  homogenous  organizations  probably  have  a  smaller  appraisal  burden 
using  vl.3 


■  Support  functions  can  reduce  the  appraisal  burden 

-  You  must  decide  which  support  functions  are  appropriate  to  declare  for  your  appraisal 

•  Gain  consensus  with  the  appraisal  team  regarding  the  use  of  support  functions  in  your  appraisal 

•  Invest  early  work  to  demonstrate  support  function  appropriateness  to  save  downstream  work 
over  the  appraisal  lifetime 

-  You  can  still  appraise  support  function  disciplines  the  “traditional”  way 


11/16/2011 
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Questions? 


Raytheon 

Missile  Systems 


11/16/2011 


Page  12 


Presenter  Biography 


Raytheon 

Missile  Systems 


28  years 
engineering 
experience 


Functions:  software,  systems  engineering,  program  management  liaison,  process  improvements 
Industries:  cell  phones,  industrial  automation,  missile  systems 
•  3  patents  -  high-reliability  systems,  redundancy,  inter-process  communication 
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Introductions 


•David  Martin 


•Some  questions  for  you 

•  SEPG 

•  Appraisers 

•  Partners 

•  Large  Organizations 

•  SME 
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Gene  Miluk 


Carnegie  Mellon 
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AIM  Implementation  Pilot 
Projects 

Presenter: 

David  Martin 


©CGI  GROUP  INC.  All  rights  reserved 


Accelerated  Improvement  Method  (AIM) 


Accelerated  Improvement  Method  (AIM) 

Integrates  and  leverages  TSP+  to  achieve  high-maturity  and  high- 
performance  with  an  agile,  “right-weight”  implementation. 


Measurement 
and  AnaVys\s 
too\k\t 


SCAMPI 


CGI 


Who  is  CGI? 


•  A  global  leader  in  IT,  business  process,  and 
professional  services,  CGI  partners  with  federal 
agencies  to  provide  end-to-end  solutions  for 
defense,  civilian,  and  intelligence  missions 

•  Acquired  Stanley  Associates,  Inc.  in  August  2010 

•  This  CGI  Business  Unit  division  has  provided 
software  services  for  our  government  customer  at 
this  site  for  over  30  years 

•  This  CGI  Business  Unit  division  has  participated 
with  its  government  customer  in  process 
improvement  since  1991. 

®CGI 
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Organizational  goals 


•  Improve  existing  software  development  processes 
and  software  team  performance 

•  Improve  software  quality 

•  Enhance  process  performance 

•  Estimations 

•  Consistency 

•  Schedule 

•  Be  prepared  to  conduct  SCAMPI  ML3  appraisal  in 
18  months  or  less 


CGI 
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Appraisal  Preparation 

Traditional  Teams 

•  Engineering  Projects 

•  Process  Group 

•  Management 

•  PPQA 

•  Org.  Support  Roles 

•  Training 

Major  impact  to  other 
functions  within  the  division 


CGI 


•  TSP  Team 

•  TSP  Projects 

•  Process  Group 

•  Management 

•  Function  Roles  (filled  by 
PG  or  TSP  Project 
Members) 

•  Minimal  Impact  on  other 
functions  within  the 
division 
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Team  A  -  Gap  Analysis  Results 
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Team  B  -  Gap  Analysis  Results 
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Organizational  -  Gap  Analysis  Results 
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•  Summary 

•  326  Adequate  Implementation  of  Mode  Practice 

•  171  Partial  Implementation  of  Model  Practice 

•  81  Implementation  Absent  or  Poorly  Addressed 


CGI 
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Gap  Analysis  Results 


•  Software  Teams 

•  Existing  processes  and  toolsets  such  as  TSP  and  version 
control  systems  added  strength  to  team  practices 

•  Many  tasks  were  being  performed  without  generating 
artifacts  necessary  for  CM  Ml 

•  Organizational  processes  are  weak 

•  Launch  the  Process  Group  as  a  TSP  Team 

•  Create  New  Organizational  Processes 

•  Track  Appraisal  Preparation  Progress 

•  Address  Identified  Weaknesses 


CGI 
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Launching  the  Process  Group 

•  Team  Composition 

•  Team  Lead,  4  additional  team  members 

•  All  working  on  a  part-time  basis 

•  252  corrective  actions  tracked  as  tasks  by  the  PG 


CGI 
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Organizational  Tailoring 


•  Organizational  processes  were  updated  to  allow  for 
TSP  to  be  used  by  software  teams  in  addition  to 
standard  software  practices 

•  TSP  Documentation  was  updated  to  reflect  CGI’s 
processes  as  they  are  practiced 


CGI 
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SCAMPI  B  Results 


CGI 
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Organizational  -  SCAMPI  B  Results 
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•  Summary 

•  574  Adequate  Implementation  of  Mode  Practice 

•  4  Partial  Implementation  of  Model  Practice 

•  2  Implementation  Absent  or  Poorly  Addressed 
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o 
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Measurement  &  analysis 
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Supplier  agreement  management 
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Project  monitoring  &  control 
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Project  planning 
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Requirements  management 

CGI 


o 

Defined 

o 

Decision  analysis  &  resolution 

o 

Risk  management 

o 

Integrated  project  management 

o 

Organizational  training 

o 

Organizational  process  definition 

o 

Organizational  process  focus 

o 
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Project  Performance  Pilot  vs  Pre-TSP 

*  Productivity  Increased  by  35% 

*  Estimated  Time  on  Task  Variance  Reduced 
from  18%  to  7% 


*  Defects  Found  in  Validation  Testing  Reduced 
by  50% 


*  Schedule  Variance  Reduced  to  Less  than  1 0% 
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Organizational 
Tailoring  of  TSP 


Processes 
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Launch 
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SCAMPI  A 
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ML  3  Rating 
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SEID  Objectives: 

-  Improve  Quality 
-Improve  Estimations 
-Improve  Productivity 
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The  Most  Pervasive  CMMI  VI  .3  Change? 


Certainly  the  most  publicized  model  changes  are  at  the  higher  maturity 
levels. 

But  consider... 

Requirements  Development ,  Specific  Practice  3.2-  Establish  and  maintain  a 
definition  of  required  functionality  and  quality  attributes. 

Quality  attributes  are  mentioned  dozens  of  times  now  throughout  the 
informative  material  of  the  model. 
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Quality  Attributes  and  Architecture 


From  the  Glossary: 

quality  attribute  -  A  property  of  a  product  or  service  by  which  its  quality  will  be 
judged  by  relevant  stakeholders.  Quality  attributes  are  characterizable  by 
some  appropriate  measure.  Quality  attributes  are  non-functional,  such  as 
timeliness,  throughput,  responsiveness,  security,  modifiability,  reliability, 
and  usability.  They  have  a  significant  influence  on  architecture. 

architecture  -  The  set  of  structures  needed  to  reason  about  a  product.  These 
structures  are  comprised  of  elements,  relations  among  them,  and 
properties  of  both. 

In  the  most  basic  sense,  quality  attributes,  whether  expressed  or  implied, 
are  what  drive  architectural  decisions. 

Put  another  way,  architecture  decisions  express  quality  attributes,  whether 
they  are  stated  or  not. 
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An  Opportunity  for  Architecture 


Background: 

•  Bolsa  Mexicana  de  Valores  (BMV) 
operates  the  Mexican  financial  markets 
under  license  from  the  federal  government. 

•  Bursatec  is  the  technology  arm  of  the 
BMV. 

•  BMV  desired  a  new  trading  engine  to 
replace  the  existing  stock  market  engine 
and  integrate  the  options  and  futures 
markets. 

•  The  BMV  performed  a  build  vs.  buy 
analysis,  and  decided  to  replace  their  three 
existing  trading  engines  with  one  in-house 
developed  system. 
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The  Project  -1 


Bursatec  committed  to  deliver  a  trading  engine  in 
8-10  quarters: 

•  High  performance  (as  fast  or  faster  than  anything  out  there) 

•  Reliable  and  of  high  quality  (the  market  cannot  go  down) 

•  Scalable  (able  to  handle  both  spikes  and  long-term  growth  in  trading  volume) 

Bursatec  approached  the  SEI  for  support  during  design  &  development. 

SETs  role — provide  methods,  techniques,  and  guidance  to  improve 
Bursatec’s  software  delivery  capability: 

•  Training  and  coaching  for  the  system  architects 

•  Training  and  coaching  for  the  development  team 
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The  Project  -2 


Architecture  Decisions  (to  satisfy  quality  attributes): 

•  Development  in  Java  (lower  TCO) 

•  Low  Latency  Communication  Multicast  Network 

•  In  memory  data  storage  during  trading  session. 

•  Hot-Hot  High  Availability  configuration. 

•  Parallel  processing  in  JVM 

•  Horizontal  scalability 

Functional  Requirements: 

•  Order  routing  with  FIX  protocol. 

•  Interconnect  to  current  legacy  systems. 

•  Combined  Cash  and  Derivatives  markets  with  a  single 
Control  Workstation. 

•  Separate  Market  Data  and  Index  calculation  system. 


T-Mjrrf 


A  Partial  List  of  Potential  Problems 


Complicating  factors: 

•  Pressure  -  managers  replaced  when  commitments  are  not  met 

•  Inexperience  -  available  staff  talented  but  young 

•  Large  project  -  scope  of  the  project  beyond  the  organization’s  recent 
experience 

•  #  of  person-months 

•  #  KLOC/fu notion  points 

•  #  of  interconnecting  platforms 

•  #  of  individual  projects 

•  Key  implementation  technologies  never  used  together  formally 

•  Constant  stream  of  new  requirements/changes  to  business  rules 


Software  Engineering  Institute  CarnegieMellon 


©2010,  2011  Carnegie  Mellon  University 


Trading  Engine  Quality  and  Other  Attributes 


Quality  Attributes 

•  Under  1ms  processing  latency 

•  Horizontal  scalability 

•  Redundant  HA  system 

•  Warm  DR  system 

•  Automatic  testing  framework 
(one  day  turnaround  attribute) 

•  Localize  business  rules  changes 
in  specific  modules 


Other  Attributes 

•  Backward  compatible  with  current 
systems 

•  Combined  platform  for  both 
markets 

•  Run  on  Commodity  hardware 

•  86  order  type/attribute 
combinations  (30  in  current 
system) 

•  Real  time  updates  to  status  of 
system  via  Control  Workstation. 
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The  Proposed  Solution  - 

Integrates  High-Value  Architecture  and  Team  Practices 


Architecture-Centric  Engineering  Team  Software  Process 


•  Proven  technology. 

•  Strongly  addresses  critical 
technical  aspects  of  the 
early  project  lifecycle  activities. 

•  Specific  focus  on  architecting  to 
meet  business  objectives. 

•  Key  managers  familiar  with 
technology  via  training  courses. 


•  Proven  technology. 

•  Strongly  addresses 
management  and  measurement 
across  the  project  lifecycle. 

•  Specific  focus  on  building  high- 
performance  teams. 

•  Key  managers  familiar  with 
technology  only  through  word- 
of-mouth  and  literature. 


TSP  has  a  large  “ out-of-the-box ”  CMMI  footprint.  Architecure  drove 
the  work  breakdown  structure  (WBS)  and  provided  a  robust 
framework  for  requirements  management. 


Software  Engineering  Institute  CarnegieMellon 


©2010,  2011  Carnegie  Mellon  University 


Architecture  Drives  the  Lifecycle 


Two  iterative  processes  based  on  the  architecture  of  the  system: 

Design  cycles  (1,  2)  Implementation  cycles  (3,  5,  6) 

The  goal  is  to  design  a  system  The  goal  is  to  implement  the 
that  ensures  business  success.  system  according  to  the  design. 
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QAW/BTW  -  Building  Quality  Attribute  Scenarios 


The  Quality  Attribute  Workshop  (QAW)  and  Business  Thread  Workshop 
(BTW) 

•  bring  together  important  internal  and  external  stakeholders 

•  develop  and  validate  key  quality  attribute  scenarios  that  quantitatively 
define  the  most  important  non-functional  requirements 

•  QAW  focuses  on  developing  quality  attribute  scenarios 

•  BTW  focuses  on  business  context  to  validate  scenarios 
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Attribute-Driven  Design  (ADD)  Method 


ADD  uses  quality  attribute  scenarios  to  drive  architectural  design. 

The  process  was  time-boxed  two  ways. 

•  Six-week  boxes  to  focus  on 

—  initial  architectural  (vl)  while  training  architect  team 

—  refined  architecture  (v2)  for  early  review  or  AT  AW 

—  “complete”  (not  final)  architecture  (v3)  for  use  by  developers2 

•  Two-week  boxes  that  focused  on 

—  developing  the  architecture 

—  preparing  for  and  performing  ATAM-based  peer-reviews  with  the 
“architecture  coach” 

1.  Development  team  was  launched  at  this  point 

2.  A  TAM  actually  occurred  at  this  point 
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Views  and  Beyond  for  Architecture  Documentation 


“View  and  Beyond  is  not  a  method,  but  a  collection  of  techniques: 

1.  Find  out  what  architecture  information  stakeholders  need. 

2.  Provide  that  information  to  satisfy  the  needs. 

3.  Capture  the  information  in  views,  plus  beyond-view  information. 

4.  Package  the  information  in  a  useful  form  to  its  stakeholders. 

5.  Review  the  result  to  see  if  it  satisfied  stakeholders’  needs.” 

From  the  SEI  class  Documenting  Software  Architectures, 
http://www.sei.cmu.edu/traininq/p33.cfm. 
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Active  Review  of  Intermediate  Designs  (ARID) 


An  ARID  was  held  in  conjunction  with  a  TSP  relaunch. 

The  purpose  of  ARID  is  to 

•  put  the  architectural  documents  into  the  hands  of  developers 

•  ensure  that  the  documents  are  fit  for  development  use  (right  information 
recorded  at  sufficient  level  of  detail) 

•  provide  early  “live”  feedback  to  the  architecture  team 
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Architecture  Trade-off  Analysis  Method  (AT AM) 


ATAM 

•  brings  together  a  system’s  stakeholders 

•  evaluates  the  existing  architecture  with  respect  to  the  quality  attribute 
scenarios 

•  focuses  on  surfacing  architectural  risks 

•  promotes  &  requires  adequate  documentation  of  the  architecture 

As  mentioned  previously,  two-day  ATAM-based  peer-reviews  were  used 
by  the  architecture  coach  during  development. 

•  on-the-job  training  for  architecture  team 

•  forced  adequate  documentation  from  the  start 

•  fewer  risks  surfaced  at  formal  ATAM  than  expected  for  size/scope  of  project 
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ACE  /  TSP  Design,  Analysis,  and  Implementation 
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Trading  Engine 


Multicast 

Network 
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Special  TSP  Roles  for  Architecture 


TSP  defines  certain  standard  roles  on  a  software  development  team. 

•  “Staff’  roles  -  planning,  quality,  process,  support 

•  “Line”  roles  -  customer  interface  (requirements),  design,  implementation, 
test 

Planning  and  performing  these  roles  have  a  large  CMMI  footprint. 

The  team  defined  three  special  roles  to  address  critical  architecture  issues. 

•  Lead  architect  -  a  coming  “standard”  role 

•  Performance  manager  -  the  #1  quality  attribute  scenarios 

•  Garbage  collection  manager  -  the  #1  technical  risk  to  performance 
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Project  History 


Cycle  1  (Architecture)  -  Completed  Jan.  2010  (on  time),  demonstrated  architecture 
coaching  for  the  first  time,  evaluation  of  comm,  packages,  built  test  framework 

Cycle  2  (Infrastructure  implementation)  -  Completed  Apr.  2010  (on  time),  included 
successful  ATAM  in  Mar.  2010  (documentation  noticeably  thorough,  no 
significant  new  architectural  risks  discovered) 

Cycle  3  (Basic  functions  and  main  performance  loop)  -  Completed  July  2010  (on 
time),  good  (not  great)  quality,  performance  exceeding  requirements  by  more 
than  a  factor  of  5 

Cycle  4  (Non-TSP  cycle,  outside  evaluation  by  world-class  experts)  -  Completed 
Aug.  2010,  JVM  &  high-speed  redundant  communications 

Cycle  5  (Full  normal  operations,  complete  performance  loop)  -  Completed  Jan. 

201 1  (on  time) 

Cycle  6  (Full  functionality  incl.  startup,  shutdown,  &  maintenance  modes)  - 
Completed  July  201 1  (additional  scope  extended  scheduled  June  finish) 
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Current  Project  Status  -  cont 


Cycle  7  -  System  Test  /  Integration  Test 

•  ALL  QUALITY  A TTRIBUTES  HA  VE  BEEN  DEMONSTRA TED  AT  OR  BETTER 
THAN  SPECIFIED  LEVELS. 

•  On  Time  (expected  Oct.  201 1  finish) 

•  Integration  Test  with  Legacy  systems 

Cycle  8  -  Acceptance  Test  /  Parallel  Test 

•  Internal  user  testing  /  certification 

•  Scheduled  to  start  in  4Q’201 1 

Cycle  9  -  User  Test  /  Deployment 

•  Brokerage  firms  testing  ,  including  functional,  HA,  throughput  and  DRP  tests 

•  Scheduled  to  start  late  201 1 

Go-Live  Scheduled  2Q’2012 
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Select  Process  Data 


Measured  size  through  cycle  7  (actual) 

•  -208  eKLOC  in  24  months 

Effort  distribution  through  cycle  6  (%  of  task  hours) 


Cycle  1 

Cycle  2 

Cycle  3 

Cycle  5 

Cycle  6 

14.4 

4.9 

19.4 

32.5 

28.8 

Effort  distribution  through  cycle  6  (%  by  “block  activities”) 


Mgt 

Req 

Arch 

DLD 

Code 

Test 

Other 

3.7 

17.5 

12.0 

18.5 

32.2 

14.5 

1.5 

25.3%  of  all  recorded  task  hours  through  cycle  6  were  some 
form  of  review  or  inspection,  48%  requirements  or  design. 
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Current  Project  Status 


•  Very  low  defect  count  in  System  Test 

•  Defects  encountered  have  not  modified  the  Architecture 

•  Unit  Test  in  place  with  high  code  coverage 

•  Testing  Framework  allowed  a  smooth  continuous  integration 

•  Regression  tests  done  within  the  same  day  (except  for  multiday  orders) 

•  Static  analysis  tools  for  Inspections  and  Architecture  Integrity 

•  Latency  and  throughput  metrics  exceeded  initial  expectations 
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Key  Takeaways 


Architecture  and  TSP  were  focused  on  core  of  the  system  (Matching 
Engine) 

Other  key  components  would  have  benefitted  with  TSP  such  as: 

•  Message  Format  translator 

•  Trading  Terminal 

Most  of  the  issues  encountered  have  been  with  the  interaction  with  legacy 
systems:  Reporting,  Billing,  Market  monitoring  due  to  legacy  fields. 

Requirements  /  Inspections  could  be  done  better  (including  DLD  interfaces 
with  legacy  systems)  to  have  a  better  defect  yield. 


=•  Software  Engineering  Institute  CarnegieMellon  ©2010,  2011  Carnegie  Mellon  University 


Future  Potential  for  TSP  &  Architecture 


This  is  not  a  complete  set  of  possible  TSP  adaptations  of  architecture 
processes. 

Applying  architecture  methods  to  a  large  legacy  system  that  requires 
significant  enhancements  demands  different  adaptations  of  the  underlying 
principles. 

Applying  SOA  (service-oriented  architecture)  methods  would  be  a  related  but 
different  set  of  adaptations. 

The  presumption  is  that  the  appropriate  combination  of  TSP  and 
architecture  methods  meets  the  intent  of  the  (new)  CMMI  practices. 


=•  Software  Engineering  Institute  CarnegieMellon  ©2010,  2011  Carnegie  Mellon  University 


Questions? 


See  the  SEI  website  for  information  on  their  architecture-related 

conference,  SATURN  2012 . 

http://www.sei.cnriu.edu/saturn/ 
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Contact  Information 


TSP  Initiative 

James  W.  Over 
TSP  Initiative  Lead 

iwo@sei.cmu.edu 

Jim  McHale 
TSP  Mentor  Coach 

idm@sei.cmu.edu 

Business  Development 


RTSS  Program 

Linda  Northrop 
RTSS  Program  Director 

lmn@sei.cmu.edu 

Felix  Bachmann 
Architecture  Mentor  Coach 

fb@sei.cmu.edu 


David  Scherb  dscherb@sei.cmu.edu 
Greg  Such  qsuch@sei.cmu.edu 

SEI  website  at  www.sei.cmu.edu  (~/tsp  or  -/architecture) 
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This  material  is  distributed  by  the  SEI  only  to  course  attendees  for  their  own  individual 
study. 

Except  for  the  U.S.  government  purposes  described  below,  this  material  SHALL  NOT  be 
reproduced  or  used  in  any  other  manner  without  requesting  formal  permission  from  the 
Software  Engineering  Institute  at  permission@sei.cmu.edu. 

This  material  was  created  in  the  performance  of  Federal  Government  Contract  Number 
FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software 
Engineering  Institute,  a  federally  funded  research  and  development  center.  The  U.S. 
Government's  rights  to  use,  modify,  reproduce,  release,  perform,  display,  or  disclose 
this  material  are  restricted  by  the  Rights  in  Technical  Data-Noncommercial  Items  clauses 
(DFAR  252-227.7013  and  DFAR  252-227.7013  Alternate  I)  contained  in  the  above 
identified  contract.  Any  reproduction  of  this  material  or  portions  thereof  marked  with 
this  legend  must  also  reproduce  the  disclaimers  contained  on  this  slide. 

Although  the  rights  granted  by  contract  do  not  require  course  attendance  to  use  this 
material  for  U.S.  Government  purposes,  the  SEI  recommends  attendance  to  ensure 
proper  understanding. 

THE  MATERIAL  IS  PROVIDED  ON  AN  “AS  IS”  BASIS,  AND  CARNEGIE  MELLON  DISCLAIMS 
ANY  AND  ALL  WARRANTIES,  IMPLIED  OR  OTHERWISE  (INCLUDING,  BUT  NOT  LIMITED 
TO,  WARRANTY  OF  FITNESS  FOR  A  PARTICULAR  PURPOSE,  RESULTS  OBTAINED 
FROM  USE  OF  THE  MATERIAL,  MERCHANTABILITY,  AND/OR  NON-INFRINGEMENT). 
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CERTIFICATE  PROGRAMS 

CERTIFICATION 

Requirements 

Software  Architecture 
Professional 

ATAM  Evaluator 

ATAM  Leader 

Software  Architecture:  Principles 
and  Practices  course 

• 

• 

• 

Documenting  Software 
Architectures  course 

• 

• 

Software  Architecture  Design 
and  Analysis  course 

• 

• 

Software  Product  Lines  course 

• 

Software  Architecture:  Principles 
and  Practices  Exam 

• 

• 

• 

ATAM  Evaluator  Training  course 

• 

• 

ATAM  Leader  Training  course 

• 

ATAM  Observation 

• 
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PSPSM  training 


Personal  Software  Process 
(PSPSM)  training  is  essential  to 
successful  TSP  implementation. 


Leading  a  Dcvclopnu 
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•  TSP  Executive  Seminar  (1  day  for  top-level  execs,  middle 
managers) 

•  TSP  Team  Leader  Training  (3  days  for  team  leads,  affected 
managers) 

•  PSP  Fundamentals  (5  days  for  software  developers) 

•  TSP  Team  Member  Training  (3  days  for  other  disciplines) 
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Building  a  High-Performance  Engineering  Team 


The  Team  Software  Process  (TSP)  is  a  development 
process  for  engineering  teams 

•  Meet  planned  commitments 

•  Produce  high-quality  products 

•  Deliver  working  software  on  time/cost 


The  TSP  provides  a  disciplined, 
measured  approach  to  engineering. 


Team  communication 
Team  Team  coordination 

Management  Project  tracking 

Risk  analysis 


Team 

Building 


Focus  on  quality,  cost,  and  schedule 
performance  by  improving  the 
management  and  engineering  of 
software  at  the  team  and  individual  level. 


Team 

Member 

Skills 


Goal  setting 
Role  assignment 
Tailored  team  process 
Detailed  balanced  plans 

Process  discipline 
Performance  measures 
Estimating  &  planning  skills 
Quality  management  skills 
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The  TSP  Launch  Process 


The  most  important  outcome  is  a  committed 
team. 
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TSP-ACE  Development  Process 
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Implementation 
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TSP  Guidelines  for  Architecture  Methods  -1 


Training  (SEI  courses  -  SAPP,  DSA,  SADA,  ESA) 

•  Software  Architecture  Principles  &  Practices  (2  days  or  11 
hrs.  online) 

•  Documenting  Software  Architectures  (2  days  -  some 
concepts  overlap  with  PSP  design  templates) 

•  Software  Architecture  Design  and  Analysis  (2  days) 

•  Evaluating  Software  Architecture  (2  days  -  can  be  replaced 
by  an  architecture  coach;  recommended  for  TSP  coaches) 
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TSP  Guidelines  for  Architecture  Methods  -2 


For  first  projects: 

•  An  architecture  coach  is  essential  for  inexperienced  teams,  replacing 
ESA  training. 

•  ESA  may  be  sufficient  for  experienced  teams,  especially  if  there  is 
architecture  expertise  elsewhere  in  the  organization. 

•  Expertise  in  defining  and  capturing  quality  attributes  (QAW)  and 
evaluating  architectures  (ATAM)  is  worth  the  price. 

Architectural  Process  Assets 

•  Views  &  Beyond  (taught  in  DSA)  informs  design  standards. 

•  ADD  (a  subject  in  SADA)  is  the  basic  architecture  design  process. 

•  Lead  Architect  is  more  than  a  design  manager. 
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v  Unfiltered  communication 
v  Environmental  factors 
-v  Input  Selection  challenges 
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Noise  via  Distortion 
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Noise  via  Mixed  Messages 
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Loop  -  Policy  and  Reporting 
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Policy  and  Reporting  -  SEPG 
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Loop  -  Planning  and  Performance 
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Objective  Evaluation 


Evaluate  -  GP  2.9 
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JestPms 


Adherence 


o  o 
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JestPns 


Reporting 


Assets,  metrics,  training 


Feedback 


Projects 
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JestPms 


Process  Group 


Skills  and  Responsibilities 


Copyright  2005,  Apogen 
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Stakeholders 


Involve  Stakeholders  -  GP  2.7 
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Functional  Groups, 
Customers, 

&  Others 


Tools  and  Resources 


Management 


Resources-  GP  2.3 
,  r  Performance  -  GP  2.8 


0 


Management 

♦  TestPros 


Management 


0 


Management 
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Specific  Practices 
(Very  Briefly) 


Specific  Practice  Examples 


v  DAR.SP1.6  Select  Solutions  “Document  and  communicate  to 
relevant  stakeholders  the  results  and  rationale  for  the 
recommended  solution.” 

v  IPM.SP2.3  Resolve  Coordination  Issues  “Communicate  issues  to 
relevant  stakeholders.” 

v  MA.SP1 .4  Specify  how  measurement  data  are  analyzed  and 
communicated;  SP2.4  Communicate  Results 

v  OPF.SP3.1  “...deployment  of  process  assets  include... identifying 
how  changes  to  organizational  process  assets  are  communicated.” 

v  OPM. SP1 .1  “quality  and  process  performance  objectives  may  need 
to  be  created  or  maintained  and  re-communicated.” 

v  PMC.SP.3.1  Monitor  Project  Risks  “Communicate  the  risk  status  to 
relevant  stakeholders.” 

v  PPQA.SP.2.1  Communicate  and  Resolve  Noncompliance  Issues 

v  RD.SP.3.2  Establish  a  Definition  of  Required  Functionality  and 
Quality  Attributes  “This  functional  description... communicates  the 
manner  in  which  the  product  will  be  used.” 
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Insights  and  Issues 


Communications  and 
Maturity  Levels 
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Issues  with  CMMI 
Application 

v  Typical  Noisy  Errors 

o  Confusing  the  model  with  process 
o  Neglecting  the  bottom  line 
o  Compartmentalization  of  improvement  efforts 
o  Maturity  Level  Mandates 

v  Common  consequence:  inability  to  measure  and  discuss 
impact  of  changes  (empty  communication) 

v  Usual  Outcome:  using  the  documentation  (medium)  as 
indicators  of  success 
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How  Issues  Arise 

v  Encoding,  decoding,  context  issues  often  are  intangible 

v  The  medium  is  tangible  and  attracts  attention 

v  The  “medium  over  message”  syndrome  surfaces 

v  Focusing  on  the  medium  can  aggravate  noise  issues, 
resulting  in  a  vicious  cycle 
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Solution:  Focus  on 
Communication 

v  The  model  implies  all  components  of  communication  loops 

v  The  model  doesn’t  tell  us  specifically  what  we  should  say 

v  The  model  does  indicate  what  we  should  be  able  to 
communicate  about 

o  Model  as  prescriptive  method  =>  noise 
o  Model  as  diagnostic  =>  real  communication  loops 
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Summary 


v  The  CMMI  describes  communications,  both  explicitly  and  by 
implication 

v  Media  is  a  critical  element,  but  not  the  only  one 

v  The  media  serves  the  messages  in  communication 

v  Improper  focus  on  the  media  may  distort  or  completely 
subvert  the  message 

v  Understanding  complete  communication  loops  and  how  the 
CMMI  relates  to  them  preserves  the  critical  messages 
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Process  Improvement  via  CMMI 


Presented  by:  OST,  Inc. 
November  17th,  2011 

<0ST 

CMMI  LEVEL  5  |  ISO  9001:2008 


Agenda 


<  Case  Study  I:  Use  CAR  to  reduce  the  time  taken  to  close 
critical  defects 

<  Case  Study  II:  Use  QPM  to  dynamically  manage  our 
program  to  optimize  expected  project  objectives 

<  Questions 


Case  Study  - 1 


Using  CAR  to  reduce  the  time  taken  to  close 

critical  defects 


Background 


►  Mission  critical  project 

►  Major  release  involving  complete  re-write  of 
350,000  lines  of  code 

►  Process  already  in  place  for  system  testing  and 
logging  of  defects 

►  Data  Source  -  Mercury  Quality  Center 

►  Daily  review  of  defects  opened,  fixed,  tested  and 
closed 
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GOAL:  Why  do  a  CAR 


►  Reduce  the  mean  and  standard  deviation  for 
the  time  taken  to  close  critical  defects 

►  Identify  the  root  causes  that  contribute  to  high 
mean  and  variance  to  close  critical  defects 
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METHOD:  Root  Cause  Analysis  Steps 


Use  l-MR  charts  to 
identify  outliers  and 
cause 


Repeat  steps  2  &  3 
for  each  identified 
phase  for  the  process 
till  all  root  causes  are 
resolved 


TOOLS:  One-way  ANOVA 


am  a 

4^1  is  I 
vv  I 


One-way  ANOVA:  Days  to  close  versus  Severity 


Q  -t- p  o  PjT  qq  vq 

Severity  2vvJJ^&n  7150 
Error  112VW^X^§  510 

Total  114vw5XJ^ 


14.01  0.000 


S  =  22.59  R-Sg  =  20.01%  =  IS.  53% 


Level  N 

Critioal^J^  43.35  33.97 

Major  72  15-22 

Minor  23vvJ&Jv^  27.43 


Individual  95%  H 

-ooled  St  Dev 

( 

( — * — ) 

( - * - ) 

■+ - - 

24  36 


Fooled  SJQ&Z  =  22.59 
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Representative  data 


TOOLS:  l-MR  Charts 


<  First  -  verify  Process  is  stable:  l-MR  charts  to  identify  outliers.  Outliers  turned  out  to 


Divided  the  analysis  into  phases  to  implement  the  correction  for  an  identified  cause, 
and  statistically  analyzed  the  results  -  iterative  process 

<  Establish  targets  at  every  phase  (Based  on  “Half  Life  metric"  developed  by  Art 

Schneiderman  -  also  referenced  in  The  Balanced  Scorecard  by  Kaplan  and  Norton) 
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*  Representative  data 


The  CAR  Process 


Brainstorming  session 
•<S  Fishbone  diagram 
<  Potential  root  causes  identified 


Representative  data 


Effort  vs  Impact  analysis  to  identify  the  priorities  of  the  actions  to 
implement 


Impact 

Effort 

High 

Medium 

Low 

Low 

© 

o 

© 

Medium 

o 

© 

o 

High 

<D 

o 

o 

Action 

Effort 

Impact 

Priority 

E.l.  Test  builds  are  not  ready  at  all  times.  There  may  be  a  delay. 

Medium 

Medium 

3 

M.l.  Train  Testers  to  classify  defect  severity  correctly 

Low 

High 

1 

M.2.  Add  priority  to  test  cases 

High 

Medium 

3 

M.3.  Ask  Testers  to  test  according  to  priority  in  functionality 

Medium 

Medium 

2 

P.l.  Add  priority  to  test  cases  according  to  Severity  (Identified  in 
Phase  3) 

Low 

High 

1 
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*  Representative  data 


Analysis(contd) 


Analyze 


<  Pareto  Charts  to  determine  the  root  causes  of  the  problem  (that  were  addressed), 
monitoring  and  recording  of  the  data  also  helped  bring  about  other  issues  that  we 
were  able  to  resolve  quickly 
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*  Representative  data 


Results 


<OST 


Cause 


Resolve 


Pareto  Chart  of  Root  Cause 


18  f 


Root  Cause 

None 

Count 

17 

Percent 

100.0 

Cum  % 

100.0 

The  mean  time  to  close  critical  defects 
is  now  at  1.6  days  and  the  standard 
deviation  is  at  2^6  -  from  a  mean  of 
25  days  and  standard  deviation  of  32 
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Representative  data 


Conclusion 


XJ 

IS 

a 


•Significant  shift  in  the  mean  and 
standard  deviation 
•Assumptions  may  not  always  be 
true 

•Statistical  data  and  analysis  = 
Quantified  information 
•Increased  confidence  in  the 
process  that  was  changed  and 
standardized  =  improved  team 
direction  and  common  goals 
•Faster  delivery  of  a  quality 
product  =  Improved  customer 
satisfaction 
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Case  Study  -  II 


Using  QPM  to  dynamically  manage  our  program 
to  optimize  expected  project  objectives 
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Project  Background 


<*  Customer 

Government  Agencies  reconciling  asset  discrepancies  between 

systems 

<  What  triggered  the  Task 

Audit  Finds  needed  to  be  addressed  and  resolved  with  a  deadline 

Task  I  -  Phase  I  Objective 

Analyze  and  Validate  the  existence  of  3660  assets  and  provide 
a  resolution 

<  Schedule 

Start  Date:  01/29/2010  -  End  Date:  04/16/2010 

<  Team  Size 

7  FTEs 


15 


Introduction 


<  What? 

Model  to  predict  quantity  of  assets  processed 

<S  Why? 

Increase  in  project  scope  by  40% 

<S  Who? 

Developed  by  SEPG  team  and  project  team 

<S  Who? 

Project  Management  and  Leadership  team 

<  When? 

Weekly 
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Business  and  Project  Goals 


◄  High  Level  Business  Goal 

►  Have  all  assets  analyzed  and  reconciled  in  time  for  the  external 
audit 

►  Credibility  of  the  client  organization  was  riding  on  the  success  of 
this  effort 

►  Divide  and  conquer  -  divide  the  work  amongst  various  teams 

<  Project  Level  Goal 

►  Have  all  assets  assigned  to  our  team  processed  to  meet  an 
internal  deadline 

►  Leave  enough  time  for  the  customer  to  review  our  analysis 
Need  for  the  model 

►  40%  increase  in  scope  at  the  50%  schedule  marker 

►  Slight  delay  in  receiving  the  assets 
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Risks  in  meeting  business  and  Project  Goals 


I 


1 


RISKS  /  IMPACTS  /  MITIGATION 


ID 

Risk 

Description 

Business  Impact 

Mitigation 

Likelihood 

Impact 

Score 

Risk 

Score 

31 

Delays  from 
AFO  to  provide 
all  w ork 
packages  by 
31212010  may 
i  mpact  OST 
schedule  to 

meet 

mmo 

milestone 

The  wort  packages  from  the 
regional  PDCs  will  be  provided  to 
OST  on  4,'T  i'201 0.  Our  initial 
understanding  was  that  these  wctx 
packages  will  be  provided  to  us  for 
review  by  03/1 2®  10.  All  work 
packages  will  still  have  to  be 
reviewed  and  leld  vetrication 
where  required  has  to  be 
completed  by  415/2910. 

DST  has  420  assets  to  process  as 
of  3/1fb2910  which  will  be  li  kel  y  be 
done  by  3/242010  at  which  point 
DST  resources  will  be  waiting  for 
work  papers  for  about  4  days  . 

*  Schedule:  415/10 
milestone  may  not  be  met 
Cost:  OST  resources  will 
not  be  working  at  foil 
capacity  which  has  a  cost 
impact 

*  AFO  Provide  total 
assets  remaining 
1,496  at  the  rate  of 
ICO  per  day 

H/210) 

High 

High 

High/ 

Red 

AFO  may 
escalate  more 
than  23  of 
assets  (1952} 

which  will 

Based  on  discussion  with  AFO, 

OST  and  AJW-21  we  estimated 
that  23  of  the  assets  not  resolved 
by  AFO  will  be  processed  and 

High  / 
Red 

33 

impact  the 
schedule,  the 
scope  and  the 
budget 
allocated  for 
the  tasks 

resolved  by  OST.  If  more  than  23 
of  the  assets  are  escalated  to 

OST  this  may  have  an  impact  on 
the  schedule  and  the  budget  of 

Task  1. 

*  Cost 

See  Next  Slides 

High 

High 

11/1/2011  S  <sosr 

1 
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Outcome(s)  Predicted  and  Stakeholder  Audience 


< 


< 


Outcome  predicted 

►  Number  of  assets  processed  (with  Confidence  Interval  and 
Prediction  Interval) 

►  Data  type  -  Continuous 
Purpose  of  the  output 

►  Determining  optimal  number  of  resources  needed  that  we  could 
propose  to  the  customer 

Stakeholders 

►  Customer  (Government  Agency) 

►  OST  resources  (Analysts) 
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Data  Collection 


<  Data  Collection: 

►  165  data  points  from  Phase  I  analysis  data 

►  All  data  analyzed  was  recorded  and  stored  using  SQL 
Server 

►  Analyzed  data  was  grouped  by  resolution  (Resolved, 
Retired,  Needs  Supporting  Documentation,  etc...) 

►  Analyzed  data  was  grouped  by  analyst 

<  Time  period  of  data  collection 

►  Baseline  is  2/18/2010  -  3/17/2010. 
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Distribution  Fitting 


^  Uu  I 


f 

A  ©risk 

-■ 

■  Define  Distribution:  J4  1-  1|P|jEBl 

Name 

Dataset  45 

S| 

Cell 

=RiskBetaGenera  fO.S7857. 1.6649. 1.29. 492. Ri5kNamel'"Dataset#5'"l) 

£l 

Formula 

1 3etaGeneral(0,87857. 1.66- 

Function 

BetaGeneral ▼] 

Parameters 

Standard 

□  1 

0.87857 

a2 

1.6649 

Min 

1 

Max 

29.492 

#| 

□Jjg  ja| 

Dataset  #5 


1.56 
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24.31 


5.0% 


@R1SK  -  Define  Distribution:  J1 


Name 

Cell 

Formula 


Dataset  44 


=R'sk  nv,?aussl'/./b/9.5./354.R  sk5h  ftl'O.OSZCi?/ j.R' skNanel'Dataset 44"  n 


|  InvGauss(7. 7679, 5. 7354,  R 


Function 

InvGauss 

Parameters 

Standard 

M 

7.7679 

A 

5.7354 

Shift 

0.032077 

ij 

jajjUljl] 

Dataset  #4 


1.17 


24.83 


[  5.0%  h| 

=lQl2<] 

0 1 


IiwGauss 

ft. 7&T$,SJ3S4:R 

(0.032077]) 


0.0321 

+  CD 

7.aaa-3 

0.0431 


a|aJe|  iaUBiel 


OK 


Cancel 


The  Model 


<  Monte  Carlo  Simulation  was  used  to  predict  the  output  (Response) 

<  @Risk  simulation  software  was  used. 

<  Output  was  calculated  using  the  function  containing  input  variables  and  baseline  data 

Model  Simulation 

=RiskCompound(El^BaseIin0s!J4) 

<  Histogram  for  the  output  is  produced  after  model  is  run  using  @Risk. 


Net  working  Days _ S 


New  Resources: 

# 

Net  New  Resource  Work  Days 

Q 

0 

Seasoned  Resources: 

# 

Net  Seasoned  Resource  Work  Days 

10 

SO 

Results _  ^ 

Number  of  Assests  Processed  (New  Resources) 

Number  of  Assests  Processed  [Seasoned  Resources) 

Total  Assets  Processed 
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The  Model 


Net  working  Days 

S 

New  Resources: 

# 

0 

Net  New  Resource  Work  Days 

0 

Seasoned  Resources: 

#  _ 10 

Net  Seasoned  Resource  Work  Days  SO 


Results 


Number  of  Assests  Processed  (New  Resources) 
Number  of  Assests  Processed  (Seasoned  Resources) 
Total  Asset;  Processed 
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Risk  Mitigation  Scenarios  presented  to  the  customer 


Scenario  1 

Scenario  2 

Scenario  3 

Description 

Meet  4/16/2010  with  increase 
Scope 

Meet  4/30/2010  deadline  wit\ 

/  increase  scope  \ 

Maintain  status  quo  with 
increase  scope 

Assumptions 

Receive  all  work  papers  by 
4/2/2010  at  rate  of  100  /  day 
starting  4/22/2010 

/  Receive  all  work  papers  by  l 
/  4/2/2010  at  rate  of  100  /  day 
starting  4/22/2010 

,  Receive  all  work  papers  by 
\  4/2/2010  at  rate  of  100  /  day 
\  starting  4/22/2010 

Action  Plan 

•  >  95%  Confidence: 

Add  9  Team  Members 

•  60  %  Confidence: 

Add  8  Team  Members 

•  6  %  Confidence: 

Add  6  Team  Members 

•95%  Confidence: 

Add  3  Team  Members 

•  85%  Confidence: 

Add  2  Team  Members 

•  10%  Confidence: 

Add  1  Team  Member 

\ 

Continue  with  current  staffing 

I  level 

Impact 

Cost 

c\st  /  Schedule  (15  days  -  April/ 
\  30th  2010)  j 

Cost  /  Schedule  (40  days  - 
May  17th  2010) 
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Results  and  Benefits 


◄  Results 

►  Increase  in  staff  was  authorized 

►  Contract  value  increased  by  50% 

►  Contract  was  changed  from  FFP  to  T&M 

►  Critical  customer  deadline  of  4/30  was  met 
•<*  Benefits 

►  The  ability  to  make  the  case  statistically  provided  analytical 
credibility 

►  Increase  in  Client  confidence  and  reputation 

►  Client  began  advertising  OST’s  modeling  capability  with  the  entire 
customer  organization 


25 


What  Worked  Well 


Excitement  about  the  potential  of  modeling 
•<*  Generating  support  for  dedicated  resources  for  modeling 
•<*  Customer  Delight 

•<S  Using  this  as  a  success  story  to  build  9  more  models 
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Summary 


Increase  in  asset  analysis  scope 

<  Creating  model  using  Montecarlo  Simulations  to  provide  confidence 
levels  with  different  staffing  numbers  and  days 

<  Multiple  scenarios  were  provided  to  the  customer  to  choose  from 

<  OST  met  the  critical  customer  deadline  of  04/30/2011 

<S  The  ability  to  make  the  case  statistically  provided  analytical  credibility 

<  Increase  in  Client  confidence  and  reputation 

<  Other  models  were  created  due  to  its  success 
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Questions 


<OST 

CMMl  l,|VE l  5  |  ISO  90m  2008 
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Thank  You 
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<OST 

CM  Ml  l,|VEL  5  |  ISO  90111.2(108 
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approaches 

Paul  D.  Byrnes 
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Mission  Solutions  Engineering  (MSE)  Process  Director 
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CMM  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  office.  CMMI  are  service  marks  of  Carnegie  Mellon  University 
SCAMPI  is  a  Service  Mark  of  Carnegie  Mellon  University 
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Agenda 


^  Brief  background  of  the  MSE  organization 
<§>  Perceived  problems  to  jointly  solve 
<§>  Targeted  improvement  areas 

■  Planning 

■  Organizational  and  data  sampling 

■  High  maturity  evidence  approach 

■  Team  composition  and  events 

■  Tooling 


This  presentation  includes  planning  activities  that  pre-dated  the  official  release  of  SCAMPI  VI.  3- 
projected  concepts  were  used  that  are  totally  in  concert  with  the  intent  of  the  method  as  released. 
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MSE  is  a  premier  supplier  of  software  and  systems  engineering  and  system  integration 
products  and  services  for  real-time  mission  critical  defense  systems 


Mission  Solutions  Engineering  (MSE)  - 

Who  We  Are 

<$>  MSE,  LLC  is  headquartered  in  Arlington,  VA  with 
operations  in  Moorestown,  NJ 

<$>  600+  personnel  /  $90M+  annual  revenue 

<$>  40+  years  of  success  in  mission  systems  development 

<$>  Originally  a  division  of  CSC,  MSE  was  acquired  by 
ASRC  Federal  Holding  Company  in  October  2010 


<$>  Domain  and  Functional  Expertise  -  Real  time  combat 
system  software  development 

■  Command  and  control  (C2) 

■  Weapons  control 

■  Display  systems 

■  Radar  control 
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■  System 
niannostics 

I  v  D 


10  Years  of  Outstanding  Process  Maturity 
21  Baselines  Evaluated  at  Level  5 


6.3.2 

7P1.7P1C 


FI  00 


DDG  1000 


BMD 

4.0.1 


Norway 


CGM 


KDX 


7P1.7P1C 


BMD 


Norway 


BMD  4.0.1 
ACB-12 


V 

K1.1 


N1.4 


* 

*  * 

*  * 


MSE 


MISSION  SOLUTIONS  ENGINEERING 


CMMI  Level  5  systems  engineering  processes  mitigate 
program  risk  and  ensure  a  culture  of  innovation. 
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Proven  Performance 


^  Delivered  Results 

■  Fielded  systems  in  over  110  surface  combatants 

■  Delivered  over  48  capability  baselines 

■  Developed  and  support  over  95  million  SLOCs 
20%  productivity  increase  since  2000 

<$>  55%  decrease  in  first  pass  test  failures  since  2001 

<$>  100%  increase  in  CPCR/STR  productivity  since 

2000 

^  50%  decrease  in  defects  for  developed  code  since 

2000 

^  80%  decrease  in  defects  for  delivered  code  since 

2000 

<$>  3%  increase  in  SPI 

■  25%  increase  in  SPI  within  1%  of  1  over  all  EVMS 
contracts 
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Problem  Statement 

MSE  had  a  long  history  of  performing  successful  SCAMPIs  (and  CBA 
IPIs  and  SCEs  before  that  dating  to  the  early  „90s).  The  legacy: 

^  Inordinate  amounts  of  effort  and  resources  preparing  for  and 

conducting  a  SCAMPI  A  (via  PHD  generation  and  very  large  teams) 

<$>  Each  appraisal  was  treated  as  an  independent  end  item  -  a  once 
every  three  years  approach  -  “regenerating”  data  and  “new”  PIIDs 

Appraisal  preparation  and  evidence  collection  was  performed 
separately  from  team  and  Lead  activities  -  not  integrated 

^  “All  or  nothing”  Class  A  rules  motivated  this  organization  to  provide 
“the  kitchen  sink”  evidence  approach  -  way  too  much. 

Old  VI. 2  rules  limited  project  participation,  and... 

^  Practice  based  appraisals  were  not  “natural”  for  a  HM  organization 

■  PIIDs  got  in  the  way  of  seeing  how  work  really  gets  done... 

■  Practice  by  practice  decomposition  artificially  fragmented  a  highly  complex 
integrated  set  of  activities. 

■  Generic  practices  coupled  with  evidence  rules  vastly  expanded  the  quantity  of 
data  required. 
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The  Goal:  Transition  to  a  More  Cost 
Effective  Effort  Profile  Without 


Time  from  appraisal  project  start  to  Time  from  appraisal  project  start  to 

beginning  of  benchmark  event  beginning  of  benchmark  event 
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An  Alternative  Solution  Implemented 

^  Used  a  more  expert  driven,  incremental,  managed  evolution  of  data 
collection  and  review  tasks  to  make  the  outcomes  both 
significantly  more  efficient  and  improved  the  quality  and  utility  of 
the  data.  Summary  approach  and  outcomes: 

■  New  sampling  rules  allowed  for  much  greater  participation 

■  Managed  Discovery  and  Phased  Data  Collection  -Incremental 
data  evolution  and  reuse  used  through  the  appraisal  lifecycle. 

■  Easier  to  collect  data  for  the  organization ,  and  easier  to  review, 
understand,  and  analyze  for  the  team 

■  Reduced  emphasis  on  PIID  preparation 

■  A  set  of  events  was  integrated  into  a  single  plan.  Used  interim 
events  to  manage  risk  and  costs 

■  Implemented  a  concept  of  “ threads ”  to  present  the  data.  HM 
data  primarily  presented  data  by  “topics,  ”  rather  than  “ practices ” 

■  Ensured  results  and  effort  expended  reflected  event  goals 
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“Real”  Organization  Sampling  Example 


Projects 

organized 

into 

Subgroups 
using  the 
sampling 
factors 


Selected  Sampling  factors 

Relative  percentages 

Selected 

Sampling 

Factor 

Project 

Customer 

Size  Location 

Large  (>zzz). 

Medium  (>yyyy<=zzz)  Foreign  or 
Small  (<=aaa)  Domestic 

Work 

type  Lifecycle  Customer 

Legacy  or  Devel  or  Customer  (A, 
New  Maint  B,  C,  D) 

Benchmarking  period 

July  2010 

SM  Feb  2011  SM 
(Actual)  (Projected) 

Subgroup 

size 

ROM  %  in 

each 

subgroup 

Notes 

A 

Large 

D 

New 

Devel 

Customer  A 

8. Ox 

5. Ox 

41% 

All  PAs 

B 

Large 

F 

New 

Devel 

Customer  B 

1.5x 

1.5x 

12% 

All  PAs 

C 

Medium 

D 

Legacy 

Maint 

Customer  C 

l.Ox 

.68x 

21% 

Levels  4-5  PAs 

start  fall  2010  end  fall 

D 

Medium 

D 

New 

Devel 

Customer  A 

Ox 

.67x 

2012 

E 

Medium 

D 

New 

Devel 

Customer  D 

.67x 

.75x 

Levels  2-3  PAs 

F 

Medium 

D 

New 

Maint 

Customer  A 

Ox 

0.5x 

Ending  fall  2010 

G 

Medium 

F 

Legacy 

Maint 

Customer  B 

.12x 

l.Ox 

23% 

H 

Medium 

F 

Legacy 

Maint 

Customer  B 

.075x 

0.5x 

1 

Medium 

F 

New 

Maint 

Customer  B 

l.Ox 

0.5x 

Smattering  of  PAs 

J 

Medium 

F 

New 

Maint 

Customer  B 

0.3x 

0.6x 

K 

Medium 

F 

New 

Maint 

Customer  B 

0.5x 

0.5x 

L 

Small 

D 

New 

Devel 

Customer  D 

0.2x 

0.15x 

Too  small.  Not  in  scope 

M 

Small 

D 

New 

Maint 

Customer  A 

Ox 

Ox 

Too  small.  Not  in  scope 

N 

Small 

F 

Legacy 

Maint 

Customer  B 

Ox 

.25x 

Too  small.  Not  in  scope 

overall  size  roughly  the 

total 

Large 

Large 

same 

High  level 
data  plan 


Company  had  defined  rules  for  large,  medium,  and  small  projects 
Company  had  defined  policy  for  what  CMMI  based  processes  were  applicable  to 
Company  would  be  considered  "large"  by  SEI  Class  A  SAS  reporting  data 
X  in  Staff  Months  (SM)  above  is  normative  to  understand  relative  size. 


Other  factors  considered  but  weren't  relevant: 
Work  Location 
Funding  Source 
Contract  type 
Application  Domain 
Functional  Area 


Other  sampling 

factors 

evaluated 


This  example  is  a  sanitized  real  example  that  was  completed  prior  to  the 
formal  SCAMPI  VI. 3  sampling  rules  being  defined. 
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“Real”  Data  Sampling  Example 

This  answers  the 


This  is  the  data 
sampling  plan 
for  the  fourth 
subgroup  on 
the  prior  slide 
(the  4th  and 
5th  color  bands 
combined  into 
one  subgroup) 


Projects 

Process  Area  Count 

3 

4 

4 

4 

3 

Process  Areas 

A 

B 

C 

D 

E 

Organizational  Process  Focus  -  OPF 

Org 

Org 

Org 

Org 

Org 

Organizational  Process  Definition  -  OPD 

Org 

Org 

Org 

Org 

Org 

Organizational  Training  -  OT 

Org 

Org 

Org 

Org 

Org 

Organizational  Process  Performance  -  OPP 

Org 

Org 

Org 

Org 

Org 

Organizational  Inovation  and  Deployment  - 

OID 

Org 

Org 

Org 

Org 

Org 

Project  Planning  -  PP  < 

X 

Project  Monitoring  and  Control  -  PMC 

X 

Supplier  Agreement  Management  -  SAM 

Org 

Org 

Org 

Org 

Org 

Integrated  Project  Management-  IPM 

X 

Risk  Management  -  RSKM 

X 

Quantiative  Project  Management  -  QPM 

X 

Requirements  Management  -  REQM 

X 

Requirements  Development  -  RD 

X 

Technical  Solution  -  TS 

X 

Product  Integration  -  PI 

X 

Validation  -  VAL 

X 

Verification  -  VER 

X 

Configuration  Management  -  CM 

X 

Process  and  Product  Quality  Assurance  -  PPQA 

X 

Measurement  and  Analysis  -  MA 

X 

Decision  Analysis  and  Resolution  -  DAR 

X 

X 

Causal  Analysis  and  Resolution  -  CAR 

X-B 

X-A 

Legend 
Level  2 
Level  3 
Level  4 
Level  5 


Subgroup:  Medium,  Foreign,  Maintenance, 
Legacy 


Maintenance 

Model 


Comments 


question,  “Now  that 
we  know  the 
subgroups  and  the 
participating 
projects,  who  will 
provide  what  data?” 


Development  SPC 


alternate  practice? 


This  sampling  plan 
assumed  that  both 
artifacts  and 
affirmations  were 
collected  from  each 
basic  unit  or 
support  function  for 
their  identified 
areas. 
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Positive  Sampling  Outcomes 


<§>  Using  sampling  factors  was  in  line  with  the  way  the 
organization  did  process  tailoring  so  it  “made  sense.” 

Some  factors  originally  thought  to  be  important  turned 
out  not  to  be.  The  SCAMPI  VI  .3  process  improved  the 
overall  organizational  analysis. 

Small  projects  always  got  “costed  out”  of  participating  in 
VI  .2  SCAMPIs.  The  new  rules  allowed  other  projects  to 
participate  -  the  Sponsors,  Projects,  and  Process 
Group  were  all  very  pleased  with  this  result. 

<§>  Increased  organizational  coverage  from: 

■  50%  of  people  in  2008  to  81%  in  201 1 

■  25%  of  projects  in  2008  to  64%  in  2011 
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Managed  discovery  is  the  baseline  preferred  approach  in  VI  .3 


Data  Collection  Approach 

<§>  Managed  Discovery  -  Used  through  a  set  of 
incremental  appraisals.  Added  additional,  smaller 
events  to  the  2010-1 1  flow  relative  to  the  2008  appraisal 
lifecycle. 

■  Class  C 

■  Appraisal  Consulting 

■  Class  B1 

■  Class  B2  (virtual)  and  B3  (virtual) 

■  Readiness  Review 

■  Class  A 


The  data  collection  plan  was  refined  after  each  event, 
requiring  less  and  less  additional  preparation  effort  and 
subsequent  team  effort. 
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Data  Collection  Plan  Improvements 

<$>  The  Data  Collection  Plan  (DCP) 

■  The  DCP  was  a  subsidiary  plan  to  the  Appraisal  Plan. 

■  The  set  of  events  noted  on  the  prior  page  (the  appraisal 
lifecycle  events)  were  all  documented  in  a  single  appraisal  plan 

■  The  plan  was  initialized  at  the  start  of  the  flow,  and  added  to 
and  evolved  over  the  duration. 

■  The  detailed  plan  data  was  maintained  in  an  excel  workbook  to 
facilitate  cohesion  and  easier  maintenance. 

■  Outputs  from  the  appraisal  tool  (Appraisal  Wizard®)  were  direct 
inputs  into  the  plan,  increasing  synergy  and  reducing  redundant 
effort. 
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Changing  scoping  complexity,  coverage  rules,  timing  of  tasks,  and  other  method 
options  requires  commensurate  updates  in  how  teams  are  formed  and  trained. 


Team  Training  and  Qualification 


^  Training 

■  Method  training  (refresher  in  this  case)  was  performed  as  a  just-in-time 
activity.  A  little  bit  before  each  event. 

■  Team  training  sessions  with  all  team  members  present  were 
performed  prior  to  each  event  (it  didn"tlook  and  feel  like  “training) 

■  Additional  training  in  CMMI  High  Maturity  practices  was  delivered. 

■  Training  in  the  designated  appraisal  tool  (Appraisal  Wizard®)  was  also 
performed  and  then  refreshed  at  each  event. 


<$>  Qualification 

■  Team  experience  excluding  the  Lead  Appraiser  far  exceeded  method 
requirements  (experience).  The  technical  approach  really  needed  this. 

■  A  qualifications  table  was  added  to  the  appraisal  plan  workbook  to 
verify  requirements  were  met 


Copyright  ©2011  ISD,  Inc. 


Increasing  efficiency  and  saving  costs  with  new  SCAMPI  approaches 


14 


Integrated 

System 

Diagnostics 

INCORPORATED 


These  approaches  were  not  really  new,  but  rather  a  more  rigorous 
approach  than  past  events  that  facilitated  being  more  efficient. 


Team  Composition 

<§>  Team  membership  changed  slightly  during  each  event.  Team 
members  (count)  changed  (reduced)  after  each  event. 

<§>  Conflicts  of  Interest  (COI)  were  explicitly  addressed  in  the  team 
qualification  section  of  the  appraisal  plan  workbook. 

■  The  Lead  only  performed  training  and  appraisal  related  tasks,  not  internal 
consulting. 

^  Affiliation:  members  were  selected  to  ensure  at  least  half  the  team 
was  totally  external  to  the  Organizational  Unit. 

■  Actually  we  exceeded  this  target. 

■  We  mixed  people  with  past  experience  with  this  OU  with  people  who  were 
“new”  to  get  fresh  perspective 

■  We  matched  up  “external”  people  with  “internal”  people  to  minimize  conflicts 

■  We  documented  mini-team  outputs  in  a  way  that  supported  adjusting  the  mini¬ 
teams  over  the  set  of  events 
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Recall  that  this  is  a  ML5  scope.  Team  sizes  relative  to  norms 
are  significantly  less  than  average  across  the  industry. 


Team  Comp  Table  by  Event 


Class  C 

Class  B 

Class  B2 

Class 

B3 

RR 

Class  A 

Organization 

Paul 

Byrnes 

Paul 

Byrnes 

Paul 

Byrnes 

Paul 

Byrnes 

Paul 

Byrnes 

Paul 

Byrnes 

ISD,  SEI  Certified  HM 
Lead  Appraiser,  Team  Lead 

Jack 

Lawrence 

Jack 

Lawrence 

Jack 

Lawrence 

Jack 

Lawrence 

ISD,  SEI  Certified  HM 
Lead  Appraiser 

Barbara 

Spence 

Barbara 

Spence 

Barbara 

Spence 

CSC  NPS  BPMO/PREMO 

Joe  Ryan 

Joe  Ryan 

Joe  Ryan 

Joe  Ryan 

Joe  Ryan 

Joe  Ryan 

MSE 

Patricia 

Brencher 

Patricia 

Brencher 

Patricia 

Brencher 

Patricia 

Brencher 

Patricia 

Brencher 

Patricia 

Brencher 

MSE 

Mel 

Wahlberg 

Mel 

Wahlberg 

CSC  NPS  BPMO  (Class  B 
and  RR  only).  Certified 
Lead  Appraiser. 

Bill 

Decker 

CSC  NPS  BPMO/PREMO 
(Class  B  only) 
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Team  Qualifications  Table 


Field  Experience  (Group  Average  6; 
Group  Total>=25) 

Project  Engineerin  Process 


Name 

Organization 

Appraisal  Role 

Management 
(GT>=10;  1 
>=6) 

g 

Manageme 

nt 

Paul  Byrnes 

ISD 

Team  Lead  -  HM 

20 

10 

21 

Jack  Lawrence 

ISD 

Team  Member  -  HM 

12 

20 

15 

Mel  Wahlberg 

CSC  NPS 

Team  Member 

28 

37 

25 

Barbara  Spence 

CSC  NPS 

Team  Member 

17 

27 

15 

Bill  Decker 

CSC  NPS 

Team  Member  -  HM 

2 

33 

15 

Joe  Ryan 

MSE 

Team  Member  -  HM 

22 

31 

15 

Pat  Brencher 

MSE 

Team  Member 

7 

24 

12 

112 

182 

118 

16.0 

26.0 

16.9 

Note:  Years  can  overlap  between  categories  -  do  not  add  up  to  total  years  -  we  are 
not  that  old!  (i.e.,  you  could  be  doing  project  management  and  process  at  same  time) 
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Other  Technical  Approach  Differences 

Documentation  links  were  entered  directly  into  the 
appraisal  database  (Appraisal  Wizard®)  and  generally 
went  to  folders  rather  than  specific  documents. 

<§>  We  made  maximum  use  of  the  90  day  clock.  Document 
review  during  the  Readiness  Review  was  done  with  a 
purpose  to  reuse  data  wherever  possible. 

■  Reviewed  documents  resulted  in  findings  (gaps)  and  we 
generated  preliminary  practice  characterizations 

■  Guidelines  for  what  could  be  reused,  and  how  re-review  of  data 
was  to  be  performed  were  documented  in  Team  Norms 

■  The  technical  approach  implemented  in  effect  started  the  main 
event  during  the  Readiness  Review. 
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Evidence  Threads 

<$>  Data  (high  maturity  in  particular)  was  presented  as 
integrated  “stories”  spanning  time  rather  than 
individual  documents  in  separate  practice  buckets. 

■  Much  more  “natural”  and  “integrated.” 

■  Much  easier  for  PHD  maintenance. 

■  Much  easier  to  see  both  legacy  and  evolution. 

■  Easier  for  the  team  to  decide  what  and  how  much  to 
review 


Microsoft  Word 
Document 
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Mapping 

matrix 


i 


Storyboard 


1 , 2,  3,  4,  5,  6,  7,8  (new  for  A) 

5 

2,  3,  4,  5,  6,  7,8  (new  for  A) 

5,  8  (new  for  A) 

Summary  Statement  (new  for  A) 

8,  Summary  Statement  (new  for  A) 

9,  (new  for  A) 


Evolution  noted  from 
subsequent  events 


V7 

Summary  Statement: 

New  for  SCAMPI  A:  The  “CR  Closure  Rate  (Dev)”  and  “CR  Closure  Rate  (Maint)”  metrics  are  defined  in  ORG  Standard  Process 
<xyz>  with  the  performance  objective  “decrease  cycle  time”.  The  objective  is  correlated  to  Strategic  Plan  Goal  number  1  -  <insert 
goal  statements  (QPM  SP  1 .1 )  The  Quality  Compliance  yield  SPC  slides  are  presented  to  management  as  the  means  of  monitoring 
the  performance  against  the  goal  and  assessing  the  QPPO.  (QPM  SP  1 .4).  One  of  the  activities  in  the  ORG  Process  Improvement 
Plan  (PIP)  is  to  expand  beyond  the  set  of  models  ORG  currently  uses  and  develop  additional  models.  The  Metrics  Group  created  a 
CR  Productivity  Model.  This  model  uses  historical  performance  characteristics  to  predict  the  number  of  CRs  that  will  be  implemented 
per  staff  month  for  the  CR  work  packages  provided  by  the  customer.... Inputs  into  the  CR  Productivity  Model  are  the  predicted  CRs 
from  the  Development  and  Maintenance  Defect  Models. 

This  thread  covers  CR  Productivity  models  of  both  development  and  maintenance  CRs.... 

Activity  Details: 

•Prior  to  November  2009  -  CR  Productivity  rate  of  <#>  CRs  per  staff  month  was  used  when  bidding  on  contracts.  (OPP  SP  1.4,  1 . 

1  .See  Project  X  Cost  Model  <date> 

•In  order  to  improve  CR  Productivity  rate  the  process  was  changed  from  random  assignment  of  personnel  to  implement  CRs  to 
maintaining  the  same  staff  from  baseline  to  baseline. 

1  .(Same  staff  charging  CR  implementation  from  one  Project  X  release  to  the  next.) 

•Statistical  Analysis  performed  on  historical  projects  to  determine  the  historical  CR  Productivity  rate.  (The  time  line  was  ordered 
based  on  the  projects"  period  of  performance.)  (OPP  SP  1 .5,  GP  2.8) 

1  .ORG  CR  Productivity  Presentation  (October  2009) 

•Statistical  Process  Control  (SPC)  performed  on  current  projects  (Pjt  X  and  Pjt  Y)  to  determine  the  current  CR  Productivity  rate 
well  as  the  stability  and  capability  of  the  process  for  the  specific  projects.  (OPP  SP  1 .5,  GP  2.8) . 


Time  line  with  direct 
links  and  mapping 
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Some  Outcomes 

Planning  started  sooner  (in  this  case,  approximately  17  months  in 
advance  of  the  Class  A  on  site  start) 

More  time  with  the  Lead  reviewing  organizational  data  to  determine  the  most  appropriate 
sample  to  meet  sponsor  objectives. 

■  More  time  allocated  to  designing  and  “right  sizing”  each  interim  event  -  appropriate 
interview  sessions  (size,  scope,  type,  etc.) 

^  Team  composition  changed  -  more  external  members,  “reuse”  of 
trained  personnel,  distributed  team  activity,  specialized  training. 


^  Managed  Discovery  shifted  the  balance  of  effort  from  site 
personnel  back  to  the  team,  but  not  terribly  so  -  the  whole 
approach  was  done  in  a  more  joint,  collaborative  manner. 


^  Automated  tooling  (Appraisal  Wizard®)  was  essential  and 
extremely  beneficial.  It  facilitated  our  ability  to  effectively  pinpoint 
issue  areas,  data  needs,  maximize  data  reuse  and  collaboration. 
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Evidence  Review  Outcomes 


<$>  Interim  appraisals  were  used  to  incrementally 
“build”  the  appraisal  database. 

<$>  Reused  appraisal  data.  Appraisal  events  and 
data  were  not  treated  as  “one-time,  ”  but  as  an 
integrated  set  leading  to  the  next. 


Characterization  and  rating  -  When  are  you 
“conducting”  the  appraisal?  It  was  on-going... 

Appraisal  Wizard®  enabled  efficient  application 
of  these  approaches. 
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Outcomes  -  Team  Conduct 


<$>  Lead  more  of  a  facilitator  rather  than  “all  knowing”  expert 

<$>  Mini-teams  with  “inside-outside”  membership  maximized 
objectivity  while  benefiting  from  “insider”  knowledge  and 

minimizing  conflicts 

^  Much  more  targeted,  parallel  effort  was  implemented 
(interview  sessions,  remote  review,  more  virtual  activity, 
greater  use  of  90-day  window). 
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Outcomes  -  Effort  and  Cost 

^  Total  cost  of  201 1  evaluation  effort  (including  the  SCAMPI  B, 
Readiness  Review,  and  SCAMPI  A)  was  9%  less  than  the  cost  of 
the  2008  events  (in  normalized  dollars.) 

<$>  Reduced  total  staff  effort  of  conducting  events  by  25%  between 
2008  and  201 1 .  This  is  the  sum  total  of  external  (ISD  and  CSC) 
consultants  and  internal  resources  for  all  events  (SCAMPI  B  &  A  in 
2008  and  SCAMPI  B,  A,  and  Readiness  Review  in  2011) 


And  this  included 
additional  events 
and  additional 
external  people!! 


Event 

Me 

ConsiAtantDays 

CSCEwAmtnr 

ConsiAtantDays  MSE  Staff  Days 

-  _  __ 

HDC  PlEfhHBiuijgi 

Staff  Marths 

2GBSCfll« 

SCAMPI  a  ass  B 

Feb-06 

20 

50 

30 

12 

SCAMPI  a  ass  A 

May-06 

10 

15 

10 

4 

TobAs 

30 

e 

40 

IS 

ZIllSCAMVAduri 

SCAMPI  a  ass  B 

Nov-10 

ID 

15 

ID 

915 

SCAMPI  RR 

Mar-11 

10 

10 

ID 

426 

SCAMPI  a  ass  A 

May-11 

14 

7 

14 

3 

TofeAs 

34 

32 

34 

16.76 
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Integrated 

System 

Diagnostics 

INCORPORATED 


Questions  and  Answers 


^  Contact  info:  Paul  D.  Byrnes,  pdbyrnes@isd-inc.com,  www.isd-inc.com 

Contact  info:  Jeff  McGarry.  Jeffrev.McGarry@missionse.com, 
www.missionse.com 
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Factors  Driving  vl.3  Changes 


NORTHROP  GRUMMAN 


•  Bring  constellations  (-DEV,  -SVC,  -ACQ)  into  harmony 

•  Simplify  the  model  architecture 

•  Reflect  modern  engineering  approaches  (Agile,  Lean  Six  Sigma, 
quality  attributes,  architectural  methods,  etc.) 

•  Clarify  high  maturity  and  enhance  its  value 
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Why  Didn't  Customers  See  Value  in  ML5? 


NORTHROP  GRUMMAN 


•  Some  adopters  viewed  high  maturity 
as  "applying  Statistical 
Process  Control  to  something" 

(i.e.,  finding  some  stable  subprocess 
with  enough  data  points  to  be  able 
to  construct  control  charts) 


Enough 
data  points 


•  Some  organizations  constructed  process  performance  baselines  and 
models  with  little  relationship  to  project  performance 

•  Some  projects  selected  subprocesses  to  quantitatively  manage  that 
did  not  contribute  significantly  to  their  customer's  quality  and 
process  objectives 
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The  Value  of  Level  4/5 
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•  The  value  of  Level  4/  5  is  insight 

-  Level  4/5  is  10-20%  cheaper  than  Level  3 
(even  though  more  is  being  done 

-  Quantitative  management  establishes 
expected  ranges  of  process  performance 

-  Process  are  stable  and  predictable  -  unusual  process  behaviors  can  be 
quickly  identified,  so  effective  corrective  action  can  be  taken 

•  To  realize  the  value  of  Level  4/  5 

-  Processes  have  to  be  stable  (performed  consistently) 

-  Processes  under  statistical  control  must  support  business  objectives 

-  Data  has  to  be  useful  and  clean 

-  Analysis  has  to  lead  to  actions 
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Understanding  the  Process 


NORTHROP  GRUMMAN 


Managing  by  Variation 


•  How  many  errors  are  typically  found  in  reviewing  an  interface  specification? 


UCL=12.30 


Mean=4.799 


LCL=-2.705 


-  Was  the  review  effective? 

-  Was  the  process  different? 

-  Is  the  work  product  different? 


Corrective  and 
preventative  actions 
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Quantitative  Project  Management 

Restructured  SPs,  clarified  quantitative  management 


NORTHROP  GRUMMAN 


PPM  VI. 2 _ 

SG  1  Quantitatively  Manage  the 
Project 

SP  1.1  Establish  the  Project's  Objectives 

SP  1.2  Compose  the  Defined  Process 

SP  1.3  Select  the  Subprocesses  that 
Will  Be  Statistically  Managed 

SP  1.4  Manage  Project  Performance 

SG  2  Statistically  Manage 

Subprocess  Performance 

SP  2.1  Select  Measures  and  Analytic 
Techniques 

SP  2.2  Apply  Statistical  Methods  to 
Understand  Variation 

SP  2.3  Monitor  Performance  of  the 
Selected  Subprocesses 

SP  2.4  Record  Statistical  Management 
Data 


0PM  Vl. 3 _ 

SG  1  Prepare  for  Quantitative 
Management 

SP  1.1  Establish  the  Project's  Objectives 

SP  1.2  Compose  the  Defined  Process 

SP  1.3  Select  Subprocesses  and 
Attributes 

SP  1.4  Select  Measures  and  Analytic 
Techniques 

SG  2  Quantitatively  Manage  the 
Project 

SP  2.1  Monitor  the  Performance  of 
Selected  Subprocesses 

SP  2.2  Manage  Project  Performance 

SP  2.3  Perform  Root  Cause  Analysis 
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Prepare  for  Quantitative  Management 
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SP  1.3  Select  Subprocesses  and  Attributes 


Select  subprocesses  and  attributes  critical  to  evaluating  performance 
and  that  help  to  achieve  the  project's  quality  and  process 
performance  objectives. 


Examples  of  criteria  used  to  select 

subprocesses  include  the  following: 

•  There  is  a  strong  correlation  with 
performance  results  that  are  addressed  in 
the  project's  objectives. 

•  Stable  performance  of  the  subprocess  is 
important. 

•  Poor  subprocess  performance  is  associated 
with  major  risks  to  the  project. 

•  One  or  more  attributes  of  the  subprocess 
serve  as  key  inputs  to  process  performance 
models  used  in  the  project. 

•  The  subprocess  will  be  executed  frequently 
enough  to  provide  sufficient  data  for 
analysis. 


Examples  of  product  and  process  attributes 

include  the  following: 

•  Effort  consumed  to  perform  the  subprocess 

•  The  rate  at  which  the  subprocess  is 
performed 

•  Cycle  time  for  process  elements  that  make 
up  the  subprocess 

•  Resource  or  materials  consumed  as  input  to 
the  sub  process 

•  Skill  level  of  the  staff  member  performing 
the  subprocess 

•  Quality  of  the  work  environment  used  to 
perform  the  subprocess 

•  Volume  of  outputs  of  the  subprocess  (e.g., 
intermediate  work  products) 

•  Quality  attributes  of  outputs  of  the 
subprocess  (e.g.,  reliability,  testability) 


7 


©Northrop  Grumman  Systems  Corporation.  All  rights  reserved. 


Quantitatively  Manage  the  Project 


NORTHROP  GRUMMAN 


SP  2.2  Manage  Project  Performance 

MonitorManage  the  project  using  statistical  and  other  quantitative 
techniques  to  determine  whether  or  not  the  project's  objectives  for 
quality  and  process  performance  will  be  satisfied,  and  identify 
corrective  action  as  appropriate 

1.  Periodically  review  the  performance  of  subprocesses. 

Stability  and  capability  data  from  monitoring  selected  subprocesses,  as 
described  in  SP2.1,  are  a  key  input  into  understanding  the  project's  overall 
ability  to  meet  quality  and  process  performance  objectives. 

4.  Use  process  performance  models  calibrated  with  project  data  to  assess 
progress  toward  achieving  the  project's  quality  and  process  performance 
objectives. 

Process  performance  models  are  used  to  assess  progress  toward  achieving 
objectives  that  cannot  be  measured  until  a  future  phase  in  the  project 
lifecycle.  Objectives  can  either  be  interim  objectives  or  overall  objectives. 
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Quantitatively  Manage  the  Project  -  2 
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SP  2.3  Perform  Root  Cause  Analysis 

Perform  root  cause  analysis  of  selected  issues  to  address 
deficiencies  in  achieving  the  project's  quality  and  process 
performance  objectives. 
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Organizational  Process  Performance 

Re-ordered  SPs,  projects  can  develop  PPBs/PPMs 


NORTHROP  GRUMMAN 


OPP  VI. 2 

SG  1  Establish  Performance 
Baselines  and  Models 

SP  1.1  Select  Processes 

SP  1.2  Establish  Process-Performance 
Measures 

SP  1.3  Establish  Quality  and  Process- 
Performance  Objectives 

SP  1.4  Establish  Process-Performance 
Baselines 

SP  1.5  Establish  Process-Performance 
Models 


OPP  Vl. 3 

SG  1  Establish  Performance 
Baselines  and  Models 

SP  1.1  Establish  Quality  and  Process 
Performance  Objectives 

SP  1.2  Select  Processes 

SP  1.3  Establish  Process  Performance 
Measures 

SP  1.4  Analyze  Process  Performance 
and  Establish  Process 
Performance  Baselines 

SP  1.5  Establish  Process  Performance 
Models 
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Establishing  Process  Performance  Baselines  MemTHnapc,ujM^ 


SP  1.4  Analyze  Process  Performance  and  Establish  Process 
Performance  Baselines 

Establish  Analyze  the  performance  of  the  selected  processes,  and 
establish  and  maintain  the  organization's  process-  performance 
baselines. 

These  baselines  are  used  to  determine  the  expected  results  of  the  process  or 
subprocess  when  used  on  a  project  under  a  given  set  of  circumstances. 

Process  performance  baselines  are  compared  to  the  organization's  quality 
and  process  performance  objectives  to  determine  if  the  quality  and  process 
performance  objectives  are  being  achieved. 

The  process  performance  baselines  are  a  measurement  of  performance  for 
the  organization's  set  of  standard  processes  at  various  levels  of  detail: 

-Sequence  of  connected  processes 

-Processes  that  cover  the  entire  life  of  the  project 

-Processes  for  developing  individual  work  products 
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Causal  Analysis  &  Resolution 

Outcomes  can  be  positive 


CAR  VI. 2 

SG  1  Determine  Causes  of  Defects 
SP  1.1  Select  Defect  Data  for  Analysis 
SP  1.2  Analyze  Causes 

SG  2  Address  Causes  of  Defects 
SP  2.1  Implement  the  Action  Proposals 
SP  2.2  Evaluate  the  Effect  of  Changes 
SP  2.3  Record  Data 
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CAR  VI  .3 

SG  1  Determine  Causes  of  Selected 
Outcomes 

SP  1.1  Select  Outcomes  for  Analysis 
SP  1.2  Analyze  Causes 

SG  2  Address  Causes  of  Selected 
Outcomes 

SP  2.1  Implement  Action  Proposals 

SP  2.2  Evaluate  the  Effect  of 
Implemented  Actions 

SP  2.3  Record  Causal  Analysis  Data 
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Selecting  Outcomes  for  Analysis 
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SP 1. 1  Select  Outcomes  for  Analysis 

Select  outcomes  for  analysis. 


Examples  of  when  to  perform  causal  analysis  include  the  following: 

•  When  a  stable  subprocess  does  not  meet  its  specified  quality  and  process  performance 
objectives,  or  when  a  subprocess  needs  to  be  stabilized 

•  During  the  task,  if  and  when  problems  warrant  a  causal  analysis  meeting 

•  When  a  work  product  exhibits  an  unexpected  deviation  from  its  requirements 

•  When  more  defects  than  anticipated  escape  from  earlier  phases  to  the  current  phase 

•  When  process  performance  exceeds  expectations 

•  At  the  start  of  a  new  phase  or  task 
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Organizational  Performance  Management 

Added  SGI,  generalized  piloting 


NORTHROP  GRUMMAN 


PPM  V1.2 


SG  1  Select  Improvements 

SP  1.1  Collect  and  Analyze  Improvement 
Proposals 

SP  1.2  Identify  and  Analyze  Innovations 

SP  1.3  Pilot  Improvements 

SP  1.4  Select  Improvements  for  Deployment 

SG  2  Deploy  Improvements 
SP  2.1  Plan  the  Deployment 
SP  2.2  Manage  the  Deployment 
SP  2.3  Measure  Improvement  Effects 


PPM  VI. 3 

SG  1  Manage  Business  Performance 

SP  1.1  Maintain  Business  Objectives 

SP  1.2  Analyze  Process  Performance  Data 

SP  1.3  Identify  Potential  Areas  for 
Improvement 

SG  2  Select  Improvements 

SP  2.1  Elicit  Suggested  Improvements 

SP  2.2  Analyze  Suggested  Improvements 

SP  2.3  Validate  Improvements 

SP  2.4  Select  and  Implement  Improvements 
for  Deployment 

SG  3  Deploy  Improvements 
SP  3.1  Plan  the  Deployment 
SP  3.2  Manage  the  Deployment 
SP  3.3  Evaluate  Improvement  Effects 
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SGI:  Manage  Business  Performance 
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SG  1  Manage  Business  Performance 

The  organization's  business  performance  is  managed  using  statistical 
and  other  quantitative  techniques  to  understand  process  performance 
shortfalls,  and  to  identify  areas  for  process  improvement. 

SP  1.1  Maintain  Business  Objectives 

Maintain  business  objectives  based  on  an  understanding  of  business 
strategies  and  actual  performance  results. 

SP  1.2  Analyze  Process  Performance  Data 

Analyze  process  performance  data  to  determine  the  organization's 
ability  to  meet  identified  business  objectives. 

SP  1.3 1  dentify  Potential  Areas  for  I  mprovement 

Identify  potential  areas  for  improvement  that  could  contribute  to 
meeting  business  objectives. 
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Double-Digit  Earnings  /  Sales  Growth 


NGC  Business  Goals  and 
Performance  Objectives 


NORTHROP  GRUMMAN 


Strategic  Objectives 


Meet  our 
commitments 


Quality  &  Process 
Performance  Objectives 


'Vigorous  pursuit 
of  new  business 


Quality  and 
process 
improvement 


Cross-sector 

collaboration 


Compete  and 
perform  as  a 
Tier  1  player 


■  Increase  understanding  of  product 
quality  across  wider  range  of  project 
processes  and  decrease  variation 

■  Develop  understanding  of  process 
performance  across  wider  range  of 
project  processes  and  decrease 
variation 


Organizational  strategic  goals  flow  down  annually 

Organizational  staff  develop  proposed  organizational 
quantitative  quality  and  process  performance  objectives 
based  on  the  strategic  goals 

The  proposed  quantitative  goals  are  reviewed  with  the 
Divisions  and  adjusted  as  needed 
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OPM  Validation 
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SP  2.3  Validate  I  improvements 

Validate  selected  improvements. 


Examples  of  validation  methods  include  the  following: 

•  Discussions  with  stakeholders,  perhaps  in  the  context  of  a  formal  review 

•  Prototype  demonstrations 

•  Pilots  of  suggested  improvements 

•  Modeling  and  simulation 


Pilots  can  be  conducted  to  evaluate  significant  changes  involving  untried,  high-risk,  or  innovative 
improvements  before  they  are  broadly  deployed.  Not  all  improvements  need  the  rigor  of  a  pilot. 
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How  Does  Level  4  &  5  Benefit  the 
Customer? 
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Organizational  Process 
Performance 


Quantitative  Project 
Management 


Organizational 

Performance 

Management 


Causal  Analysis  and 
Resolution 


c 


More  accurate  estimates 


Problem  behaviors  are 
recognized  faster,  enabling 
quicker  resolution 


The  organization  and  projects 
benefit  from  improvements 
found  and  proven  on  other 

pr°iects . . f . 

The  project  fixes  the  source  of 
defects  to  prevent  future 
defects 


Adapted  from  “How  Does  High  Maturity  Benefit  the  Customer?”,  R.  Hefner,  Systems  &  Software  Technology  Conference,  2005 
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Background 
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•  Most  CMMI  adopters  continue  to  write  plans  the  traditional 
way  --  hundreds  of  pages  long,  filled  mostly  with  boilerplate 

•  This  approach  is  not  consistent  with  the  CMMI  model,  and 
makes  the  plans  difficult  (and  time-consuming)  to  create, 
use,  and  maintain 

•  This  presentation  will  describe  a  simple,  easy  to  use 
method  for  creating  a  short  (less  than  10  pages),  effective, 
CMMI-compliant  project  plan 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Topics 
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•  CMMI  planning  practices  (PP,  IPM,  GP  2.2) 

•  Policy,  plans,  process  descriptions,  procedures  -  what’s 
the  difference? 

•  A  10-page  (or  less)  planning  template 

•  Lessons  learned 
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CMMI  Planning  Practices 
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•  Planning  is  mentioned  both  as  a  process  area  and  as  a 
generic  practice  in  all  process  areas 


Requirements  Management 

CD 

C 

'c 

c 

03 

Q_ 

-*—> 

o 

CD 

e1 

CL 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organization  Process  Focus 

Organization  process  definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Organizational  Process  Performance 

Quantitative  Project  Management 

Organizational  Performance  Management 

Causal  Analysis  and  Resolution 

GP  2.1 

Establish  an  Organizational  Policy 

a 

GP  2.2 

Plan  the  Process 

GP  2.3 

Provide  Resources 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Control  Work  Products 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10 

Review  Status  with  Higher  Level  Management 

GP  3.1 

Establish  a  Defined  Process 

GP  3.2 

Collect  Process  Related  Experiences 

11 
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What  Might  Be  Included  in  a  Plan? 
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GP  2.2  Plan  the  Process 

Establish  and  maintain  the  plan  for  performing  the  process. 


The  plan  for  performing  the  process  typically  includes  the  following: 

•  Process  description  GP  3.1 

•  Standards  and  requirements  for  the  work  products  and  services  of  the  process 

•  Specific  objectives  for  the  execution  of  the  process  and  its  results  (e.g.,  quality,  time 
scale,  cycle  time,  use  of  resources) 

•  Dependencies  among  the  activities,  work  products,  and  services  of  the  process 

•  Resources  (e.g.,  funding,  people,  tools)  needed  to  perform  the  process 

•  Assignment  of  responsibility  and  authority 


GP  2.3 


GP  2.4 


Training  needed  for  performing  and  supporting  the  process 
Work  products  to  be  controlled  and  the  level  of  control  to  be  applied 

Measurement  requirements  to  provide  insight  into  the  execution  of  the  process,  its  work 
products,  and  its  services 

Involvement  of  relevant  stakeholders 

Activities  for  monitoring  and  controlling  the  process 

Objective  evaluation  activities  of  the  process 

Management  review  activities  for  the  process  and  the  work  products 
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CMMI  Planning  Process  Areas 
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Project  Planning 

SG  1  Establish  Estimates 
SP  1.1  Estimate  the  Scope  of  the  Project 
SP  1.2  Establish  Estimates  of  Work  Product 
and  Task  Attributes 
SP  1.3  Define  Project  Lifecycle  Phases 
SP  1.4  Estimate  Effort  and  Cost 

SG  2  Develop  a  Project  Plan 
SP  2.1  Establish  the  Budget  and  Schedule 
SP  2.2  Identify  Project  Risks 
SP  2.3  Plan  Data  Management 
SP  2.4  Plan  for  Project  Resources 
SP  2.5  Plan  Needed  Knowledge  and  Skills 
SP  2.6  Plan  Stakeholder  Involvement 
SP  2.7  Establish  the  Project  Plan 

SG  3  Obtain  Commitment  to  the  Plan 
SP  3.1  Review  Plans  that  Affect  the  Project 
SP  3.2  Reconcile  Work  and  Resource  Levels 
SP  3.3  Obtain  Plan  Commitment 


I  ntegrated  Project  Management 

SG  1  Use  the  Project's  Defined  Process 
SP  1.1  Establish  the  Project's  Defined 
Process 

SP  1.2  Use  Organizational  Process  Assets 
for  Planning  Project  Activities 
SP  1.3  Establish  the  Project's  Work 
Environment 
SP  1.4  Integrate  Plans 
SP  1.5  Manage  the  Project  Using  Integrated 
Plans 

SP  1.6  Establish  Teams 
SP  1.7  Contribute  to  Organizational  Process 
Assets 

SG  2  Coordinate  and  Collaborate  with 
Relevant  Stakeholders 
SP  2.1  Manage  Stakeholder  Involvement 
SP  2.2  Manage  Dependencies 
SP  2.3  Resolve  Coordination  Issues 
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Other  CMMI  Planning  Implications 
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Words  like  “selected”,  “identify”,  in  practices  imply  a  choice  to 
be  made  in  planning 

GP  2.6  Place  selected  work  products  of  the  process  under 
appropriate  levels  of  control. 

GP  2.7  Identify  and  involve  the  relevant  stakeholders  of  the 
process  as  planned. 

PPQA  SP  1 .1  Objectively  evaluate  selected  performed 
processes  against  the  applicable  process  descriptions, 
standards,  and  procedures. 
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How  do  Plans  and  Process  Descriptions  Differ? 
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Plan 

•  Description  of  activities 

•  Resources  (including 
funding,  people,  and  tools) 

•  Schedule 

•  Assignment  of  responsibility 
and  authority 


GP  2.3 


GP  2.4 


At  Level  2,  plans  describe  what  to  do 

At  Level  3,  the  existence  of  a  process 
description  means  plans  become 
much  shorter 

■  Focus  is  on  instantiating  the 
process  (e.g.,  how  often  a 
process  executes) 


Process  Description  GP  3.1 

•  Process  roles 

•  Applicable  standards 

•  Applicable  procedures,  methods, 
tools,  and  resources 

•  Process  performance  objectives 

•  Entry  criteria 

•  Inputs 

•  Verification  points  (e.g.,  peer 
reviews) 

•  Outputs 

•  Interfaces 

•  Exit  criteria 

•  Product  and  process  measures 
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A  Top-Level  Comparison 


NORTHROP  GRUMMAN 


Policy 


High-level  “what"  to  do 
(organizational  guidance) 


Process  High-level  “how"  to  do 

(organizational  standard,  tailored  by  projects) 


Procedure  Low-level  “how"  to  do 

(details  needed  to  follow  a  strategy) 


I  nstantiation  of  the  process 
(how  often,  when,  etc.) 
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Documenting  Choices  in  Plans 


NORTHROP  GRUMMAN 


•  Policies  identify  what  must  happen 

•  Process  descriptions  and  procedures  describe  the  steps  to  be  performed 

•  Plans  describe  how  the  process  is  instantiated 


Policy 


Process 


Plan 


The  fence  shall  be 
painted  each 
spring. 


1 .  Wash  fence 

2.  Sand  fence 

3.  Apply  primer 

4.  Apply  paint 

5.  Clean  brushes 


Rick 

Saturday  morning 
Fine  sandpaper 
White  paint 
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Procedure:  Clean  brushes 

1.  Xxxxxx 

2.  Xxxxx 

3.  Etc.  r. 


Bottom  Line 


NORTHROP  GRUMMAN 


•  For  Maturity  Level  3  and  higher  organizations,  the 
existence  of  a  process  description  means  the  typical 
“boilerplate”  process  descriptions  included  in  a  plan  (e.g., 
DOD-STD-2167A)  can  be  eliminated 

•  Plans  simply  describe  the  instantiation  of  the  process 

-  Who,  how  often,  what  resources 

•  Plans  capture  the  decisions  about  how  to  best  fit  the 
process  to  the  task  at  hand 

-  By  creating  a  short,  table-based  template,  the  decisions  are 
highlighted 
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Plan  Template 
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The  work  products  to  be  controlled  for  each  project  process,  their  level  of  control,  and  the 
control  authority  authorized  to  make  changes  to  the  work  product  are  defined  in  Table  2.10. 
The  levels  of  control  are  defined  in  Table  2.1 1 


Table  2.10.  Work  Products .  Documents .  and  Records 


PROJECT  PROCESS 

WORK  PRODUCT 

LEVEL  OF  CONTROL 

AUTHORITY 

ProjectPlanning 

Project  Management  Plan 

Project 

ProjectManager 

Engineering  Change  Proposals 
(CDRLA017) 

Project 

ProjectManager 

Project  Monitor  and  Control 

Work  Breakdown  Structure 

Project 

ProjectManager 

Technology  Control  Plan  (CDRL 
AQ01} 

Project 

ProjectManager 

Contract/Funds  Status  Report 
(CDRL  AO  03) 

Project 

ProjectManager 

Cost-Schedule  Status  Report 
(CDRL  AO  04} 

Project 

ProjectManager 

•  All  CMMI-required  decisions  are  captured  in  the  template 

•  The  template  can  include  either  blanks  to  fill  in,  “typical”  values  to  be 
reviewed/modified  as  needed,  or  mandatory  values  set  by  the  organization 
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Lessons  Learned 


NORTHROP  GRUMMAN 


•  The  template  has  been  a  very  useful  tool  for  explaining 
CM  Ml  concepts 

•  The  table  approach  encourages  Project  Managers  to  be 
more  conscientious  in  their  process  decisions 

•  Shorter  plans  are  easier  to  create  -  helps  CMMI  buy-in 

•  Shorter  plans  are  easier  to  read  -  no  more  shelfware 
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Background 


NORTHROP  GRUMMAN 


•  Once  they’ve  passed  their  first  appraisal,  many  organizations  have 
difficulty  maintaining  CMMI-compliant  behavior 

-  Projects  regress  into  old  habits 

-  Senior  management  turns  their  attention  to  other  challenges 

-  None  of  the  investment  pays  off 

-  When  the  time  comes  to  re-appraise,  you  have  to  start  all  over 

This  presentation  will  examine: 

•  How  to  sustain  CMMI-compliant  behavior  across  an  organization 

-  Why  projects  and  organizations  fail  to  institutionalize  CMMI  practices, 
and  ways  to  overcome  theses  problems 

•  How  evidence  gathering  differs  in  a  re-appraisal 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Agenda 
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•  What  institutionalization  is  -  and  how  to  spot  it 

•  Why  institutionalization  fails 

-  Understanding  your  organization’s  culture 

-  Why  quality  assurance  is  key  (and  why  most  QA  efforts  are  focused  on 
the  wrong  things) 

-  Resources  required  to  sustain  maturity 

-  Keeping  senior  management  support 

•  How  evidence  gathering  differs  for  a  re-appraisal 

-  What  evidence  needs  to  be  “refreshed” 

-  Common  areas  of  recidivism 
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What  is  Institutionalization? 
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Institutionalization:  The  ingrained  way  of  doing  business  that  an  organization 
follows  routinely  follows  as  part  of  its  corporate  culture. 


-  CMMI-DEV  vl .3 


When  mentioned  in  the  generic  goal 
and  generic  practice  descriptions, 
institutionalization  implies  that  the 
process  is  ingrained  in  the  way  the 
work  is  performed  and  there  is 
commitment  and  consistency  to 
performing  the  process. 

An  institutionalized  process  is  more 
likely  to  be  retained  during  times  of 
stress. 


GG  2  Institutionalize  a  Managed  Process 

GP  2.1  Establish  an  Organizational  Policy 

GP  2.2  Plan  the  Process 

GP  2.3  Provide  Resources 

GP  2.4  Assign  Responsibility 

GP  2.5  Train  People 

GP  2.6  Control  Work  Products 

GP  2.7  Identify  and  Involve  Relevant 
Stakeholders 

GP  2.8  Monitor  and  Control  the  Process 
GP  2.9  Objectively  Evaluate  Adherence 

GP  2.10  Review  Status  with  Higher  Level 
Management 

GG  3  Institutionalize  a  Defined  Process 

GP  3.1  Establish  a  Defined  Process 
GP  3.2  Collect  Process  Related  Experiences 
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Understanding  Your  Organization’s  Culture 


•  Few  engineers  or  managers  are  trained  in  organizational  psychology 

•  Improvement  efforts  implement  the  generic  practices  (i.e.,  change  the 
artifacts)  without  understanding  or  addressing  lower  level  contributors 
to  culture 


Covert  level 


Overt  level 


Intermediate 


Opinions — Behavior — Conduct — Do  &  Don’ts 
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Common  Features  - 
A  Lost  Perspective  in  CMMI  vl.2  and  3! 
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Commitment  to  Perform 
GP  2.1  Establish  an  Organizational  Policy 


V _ _ _ J 


Ability  to  Perform 
GP  2.2  Plan  the  Process 
GP  2.3  Provide  Resources 
GP  2.4  Assign  Responsibility 
GP  2.5  Train  People 
GP  3.1  Establish  a  Defined  Process 


f  'N 

Verifying  Implementation 
GP  2.9  Objectively  Evaluate  Adherence 
GP  2.10  Review  Status  with  Higher  Level 
Management 

V _ _ _ J 


Directing  Implementation 
GP  2.6  Control  Work  Products 
GP  2.7  Identify  and  Involve  Relevant 
Stakeholders 

GP  2.8  Monitor  and  Control  the  Process 
GP  3.2  Collect  Process  Related  Experiences 
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Five  Dimensions  of  Work 

Reference:  Richard  Hackman  &  Greg  Oldham,  Work  Redesign 
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•  Skill  variety  -  The  degree  to  which  the  work 
requires  you  to  exercise  a  variety  of  skills 

•  Task  identity  -  The  degree  to  which  the 
work  requires  you  to  complete  a  whole, 
identifiable  piece  of  work 

•  Task  significance  -  The  degree  to  which 
your  work  affects  others  and  contributes  to 
social  welfare 

•  Autonomy  -  The  degree  to  which  you  have 
control  over  the  means  and  methods  you 
use  to  perform  your  work 

•  Job  feedback  -  The  degree  to  which 
carrying  out  the  work  itself  provides  you 
with  direct  and  clear  information  about  how 
effective  you  are. 
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Perceptions  of  the  CMMI  Common  Features 
Based  on  Work  Environment  Preferences 
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Commitment  to  Perform 

Establish  an  Org.  Policy 

Ability  to  Perform 

Plan  the  Process 
Provide  Resources 
Assign  Responsibility 
Train  People 

Establish  a  Defined  Process 

Directing  Implementation 

Control  Work  Products 
Identify/Involve  Stakeholders 
Monitor/Control  the  Process 
Collect  Process  Experiences 

Verification 

Obj.  Evaluate  Adherence 
Review  with  Higher  Mgmt 


Skill 

Variety 


Task 

Identity 


Task 

Significance 


Autonomy 


Job 

Feedback 


“Aligning  CMMI  Strategies  with  Individual,  Project,  and  Organizational  Perspectives,  ”  Software  Technology  Conference,  2003 
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Why  QA  is  Key 
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•  Process  and  product  audits  provide  tangible, 
objective  measures  of  adoption/sustainment 

-  Policies,  processes,  and  standards  must  reflect  the 
desired  behaviors 

•  Appraisals  evaluate  the  effectiveness  of  the  audit  program 

-  Standardized  tools,  approaches,  and  methods 

-  Consistency  of  appraisers  -  if  they  understand  the  way  we  are  structured 
and  operate,  there  is  less  time  required  to  understand  what  we  are  doing. 

-  Pre-appraisal  activities  to  prepare  projects  for  the  appraisal  process 

•  The  frequency  of  audits  and  appraisals,  and  the  sampling,  must  reflect  the 
progress  of  the  cultural  change 

-  As  the  culture  begins  the  change,  more  frequent  and  more  in-depth 
audits/appraisals  are  required 

-  Later,  the  amount  of  audits/appraisal  may  decrease,  if  the  culture  has  truly 
changed 
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The  Problem  with  Traditional  QA 
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Traditional  QA 
(DoD-STD-2168) 


CMMI  Approach 


Involved  checking  documents 
against  DIDs  (data  item 
descriptions 

PPQA  emphasizes  both  work 
product  and  process  audits 

No  DIDs  means  PMs  must  decide 
what  work  products  need  standards 

Some  organizations  performed 
process  audits  against  plans 

Process  audits  are  performed 
against  process  descriptions  and 
procedures 

Criteria  were  often  subjective 

Criteria  are  defined  in  the 
procedures  and  work  product 
standard 
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QA’s  Role  Evolves 


Initially,  effective  QA 
enforces  the  new  processes 

-  Conflict  exists  when  QA  is  also 
in  the  process  improvement  role 
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Level 

5  Optimizing 

Focus 

Continuous 

process 

improvement 

Process  Areas 

Causal  Analysis  and  Resolution 

Organizational  Performance  Management 

4  Quantitatively 
Managed 

Quantitative 

management 

Quantitative  Project  Management 

Organizational  Process  Performance 

3  Defined 

Process 

standardization 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

2  Managed 

Basic 

project 

management 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 
Configuration  Management 

1  Performed 

At  higher  maturity  levels,  the  need 
for  traditional  QA  often  diminishes 

-  QA  personnel  measures 
effectiveness,  may  perform 
metrics/statistics  collection  and 
analysis 
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Resources  Needed: 

Management  Commitment  and  Support 
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Committed  management: 

•  Understands  the  key  messages 

•  Is  willing  to  take  actions  to  reinforce  them 

•  Provides  resources  to  support/sustain  process  improvement  efforts 

•  Sets  expectations  that  essential  project  functions  will  be  funded  and 
processes  will  be  followed 

-  Project  planning,  estimation,  tailoring,  CM,  QA,  etc. 

•  Supports  process  improvement  and  sustainment,  rather  than  passing 
appraisals 

•  Rewards  following  the  agreed-to  processes  rather  than  individual 
heroics 

-  “Tell  me  how  you  will  reward  me,  and  I’ll  tell  how  I  will  behave” 
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Accountability 
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•  Adopting  and  sustaining  CMMI  is  about  each  practitioner  learning  and 
performing  the  new  behaviors 

•  The  role  of  management  in  cultural  change  is  to  hold  people 
accountable  for  the  new  behaviors  and  conduct 

•  Change  agents  can  enable  management  by: 

-  Helping  them  have  a  clear  vision  of  the  new  culture 

-  Identifying  inappropriate  behavior 

-  Providing  tangible,  objective 
measures  of  adoption/sustainment 


"Crossing  The  Chasm“,  Geoffrey  Moore 
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How  Evidence  Gathering  Differs  in  a 
Re-appraisal 
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In  a  first  appraisal,  evidence 
gathering  is  typically 
performed  as  part  of  an 
initial  gap  assessment 

-  CMMI  awareness  and  buy-in, 
education,  assessment 


•  In  a  re-appraisal,  evidence 
gathering  should  focus  on 
assessing  the  effectiveness 
of  institutionalization 

-  Are  processes  in  place? 

-  If  not,  why?  What  GPs  are 
ineffective? 


•  New  projects  in  a  re¬ 
appraisal  will  have  a  similar 
focus  to  an  initial  appraisal 

-  Provides  an  assessment 
of  how  effective  your  start¬ 
up  processes  are 
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Re-appraising  Continuing  Projects 

Can  I  use  3-year  old  evidence? 
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Possibly  good 

-  PP,  GP  2.2,  GP  3.1  (need  to  check  whether  plans,  tailoring  are  being 
maintaining  current  with  program  scope) 

-  “Prepare  for”  Specific  Goal  1  ’s  (confirm  strategies  have  not  changed) 


Probably  bad 


-  PMC,  GP  2.8,  GP  2.10,  IPM,  GP  2.7,  RskM,  QPM 


-  CM,  GP  2.6,  QA,  GP  2.9,  MA,  DAR,  CAR  (use  more  recent  application) 

-  RM,  RD  (unless  no  requirement  changes) 

-  TS,  PI,  VER,  VAL  (all  driven  by  requirement  changes) 

-  OPF,  OPD,  OT,  OPP,  OPM,  GP  2.1,  GP  2.5,  GP  3.2 

'  It  depends 

-  GP  2.3,  GP  2.4 
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Summary 
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•  The  difficulty  in  passing  a  re-appraisal  is  directly  related  to  how 
effectively  practices  have  been  institutionalized 

-  Depends  on  the  strength  of  the  Generic  Practices  (Common  Features), 
the  commitment  of  management,  and  the  role  of  QA 

•  Re-appraisal  offer  a  mechanism  to  assess  the  effectiveness  of  the 
generic  practices 

-  Focus  should  be  preventing  recidivism,  not  just  correcting  the  evidence 
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Background 
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•  Convincing  someone  to  implement  a  CMMI  practice  is 
especially  difficult  when  they  don’t  believe  the  practice  adds 
any  value 

•  It  is  important  that  CMMI  advocates  be  able  to  clearly  explain 
the  business  value  of  each  practice  and  its  positive  impact  on 
project  and  organization  performance 

•  This  presentation  will  outline  the  potential  benefits  from 
adopting  CMMI  practices 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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What  is  CMMI? 
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•  The  CMMI  is  a  model  of  industry  best-practices  for 
engineering  products 

•  When  an  organization  decides  to  adopt  CMMI,  they  commit  to 
performing  these  best-practices 

-  Different  than  a  customer-driven  process,  where  you  simply  do  what 
the  customer  asks  you  to  do 

•  You  are  performing  practices  in  what  you  understand  to  be 
the  best  way  known  in  industry 

-  “Best”  implies  predictably  producing  products  of  acceptable  quality  at 
the  lowest  possible  cost  and  schedule 
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Adopting  the  CMMI 
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CMMI  practices 


Already 

performing 


Not  performing 


Determine  how 
best  to  perform 
and  document 
t _ 


Not  aware  of 


Aware  of 


Perceive  as  Don’t  perceive  as 
valuable  valuable 


Understand  the 
value 


Key  enablers 

Willingness  to  learn  unfamiliar  practices 
Desire  to  extract  value  not  “check  the  box” 
Ability  to  interpret  the  CMMI  in  your  context 
Understanding  the  value  of  the  CMMI  practices 
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A  Typical  Interchange 


CMMI  Change  Agent 


“You’re  not  doing 
practice  X.” 

“You  must  do  that  practice 
to  satisfy  CMMI!” 

“Practice  X  adds  value” 

“Well,  it’s  in  the  CMMI, 
so  it  must  be  important.” 

“Well...,  you  have  to  do  the 
practice  or  you’ll  fail  the  appraisal!” 


NORTHROP  GRUMMAN 


CMMI  Change  “Target” 

■% 

-SO-  fl 

“The  customer  didn’t  say 
we  have  to  do  practice  X.” 

“How? 

“Practice  X  doesn’t  make  sense 
for  us  -  we’re  special.” 

“$A&*&%!!!!!” 
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Underlying  Principles  of  CMMI 
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•  Process  discipline  leads  to  predictable  project  performance 

-  Say  what  you  do;  do  what  you  say 

-  Document  the  plans/processes 

-  Communicate  them  to  the  performers  and  stakeholders 

-  Audit  to  ensure  we  are  following  them 

•  Conscious  choices  lead  to  better  processes 

-  E.g.,  identify  relevant  stakeholders  and  their  involvement;  identify  work 
products  to  be  controlled  and  the  control  method;  define  validation 
procedures  and  criteria,  ... 

•  Organizational  learning  improves  project  performance 

-  Capture  what  works,  and  what  doesn’t 

-  Make  rules  (policies)  to  guide  projects 

-  Define  expected  processes,  and  let  projects  tailor  them  to  fit 

-  Capture  work  products  and  measures,  and  learn  from  them 

“Interpreting  the  CMMI:  It  Depends!”,  R,  Hefner  and  S.  Yellayi, 
2005  CMMI  Technology  Conference  and  User  Group 
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How  Do  the  CMMI  Practices  Add  Value 
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•  Each  practice  provides  value  in  4  possible  ways: 

-  Performance  -  the  practice  directly  reduces  cost  and  or  schedule 
through  either  increased  efficiency,  increased  effectiveness,  or 
lowered  rework 

-  Quality  -  the  practice  produces  higher  quality  products,  by  either 
preventing  or  uncovering  defects 

-  Insight  -  the  practice  helps  explain  current  process  or  project 
behaviors,  enabling  action 

-  Communications  -  the  practice  helps  everyone  understand  expected 
behavior,  or  provides  insight  leading  to  better  decisions 

•  Many  practices  effect  more  than  one  dimension 

•  Some  practices  provide  the  potential  for  a  positive  impact  or 
reduce  the  risk  of  a  negative  impact 


7 


©Northrop  Grumman  Systems  Corporation.  All  rights  reserved. 


Some  CM  Ml  Areas  Offer  More  Potential 
Value  than  Others 
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•  The  activities  which  drive  cost  and  schedule  the  most  provide 
the  most  potential  for  productivity  improvement 

•  For  most  large  companies  and  large  software  projects,  the  most 
expensive  and  time  consuming  activities,  in  rank  order  are: 

-  Defect  removal 

-  Producing  documents 

-  Meetings  and  communications 

-  Coding 

-  Project  management 


“The  Schedules,  Costs,  and  Value  of  Software  Process 
Improvements,”  Caper  Jones,  2007,  used  with  permission 
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Why  Do  People  Resist  Change? 
Perceived  Loss  of  Personal  Power 


NORTHROP  GRUMMAN 
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Barriers  to  Seeing  the  Value 
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“Sometimes  you  have  to  believe  it  to  see  it.” 

•  Practitioners  may  not  have  worked  in  an  environment  where 
the  practice  was  performed 

•  Practitioners  may  have  worked  in  an  environment  where  the 
practice  was  performed  poorly  or  in  a  non-value-added 
manner 

•  The  practice  may  run  counter  to  a  long-held  belief 

•  Believing  the  practice  is  an  improvement  may  require  an 
action  the  practitioner  is  not  willing  to  take 

-  Awkwardness  of  doing  something  new 

-  Admit  they’ve  been  doing  it  wrong 

-  Loss  of  personal  power  when  perceived  to  be  an  expert  in  the  current 
approach 
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Some  CMMI  Areas  Where  the  Value 
Isn’t  Obvious  to  Many  Practitioners 


NORTHROP  GRUMMAN 


•  PP  SP  1 .2  Establish  Estimates  of  Work  Product  and  Task  Attributes 

-  PMC  SP  1.1  Monitor  Project  Planning  Parameters 

•  Process  and  Product  Quality  Assurance 

•  Risk  Management 

•  Decision  Analysis  &  Resolution 

•  Supplier  Agreement  Management 

•  GP  2.6  Control  Work  Products 

-  PMC  SP  1 .4  Monitor  Data  Management 

•  GP  2.7  Identify  and  Involve  Relevant  Stakeholders 

-  PMC  SP  1 .5  Monitor  Stakeholder  Involvement 

•  Level  4/5 
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Planning 


NORTHROP  GRUMMAN 


•  The  value  of  planning  is  communication 

-  Plans  provide  a  structured  way  to  investigate 
the  best  approach  for  satisfying  given  objectives 

-  Practitioners  know  what  is  expected  of  them, 
so  fewer  mistakes  are  made,  avoiding  rework 

-  Interdependences  are  identified  and  negotiated  which  causes  fewer 
disconnects  between  groups,  avoiding  schedule  delays  and  wasted 
resources 

-  We  monitor  progress  against  plans,  so  that  corrective  actions  can  be 
taken  more  quickly  and  insightfully 

•  To  realize  the  value  of  planning 

-  The  environment  has  to  be  predictable 

-  Plans  must  be  reasonable 

-  Plans  must  be  up-to-date 

-  Plans  must  be  communicated  to  those  involved  and  followed 

-  The  process  behind  the  plans  must  be  communicated  and  followed 
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Planning  Example 


NORTHROP  GRUMMAN 


Project  Planning  -  Specific  Practice  1.2 

Establish  and  maintain  estimates  of  work  products  and  task  attributes. 


Measures  of  size  of 
complexity  that  contribute  to 
determining  effort  and  cost 


•  Frequently  heard: 

-  “It’s  not  important  to  estimate  or  track  code  size,  it  changes  anyway” 

-  “We  don’t  use  attributes,  we  just  know  it’s  a  X  week  effort” 

•  Explaining  the  value 

-  Everyone  uses  attributes,  perhaps  subconsciously  -  we  just  want  to 
make  it  conscious 
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Risk  Management 


NORTHROP  GRUMMAN 


•  The  value  of  risk  management  is  performance 

-  By  identifying  potential  risks  and  mitigating  them, 
you  prevent  schedule  delays  caused  by  a  problem 

-  Without  risk  management,  effectiveness  is  hindered 
because  events  outside  your  control  un-do  your 
careful  planning  and  execution 

-  Must  accept  that  some  mitigation  actions  will  be 
“wasted”  because  the  risk  is  never  realized 


•  To  realize  the  value  of  risk  management 

-  The  culture  must  support  an  open  discussion  of  risks 

-  Must  be  effective  at  identifying  ALL  risks,  ALL  perspectives 

-  Risk  mitigations  must  offer  a  reasonable  return  on  investment 

-  Solid  plans  must  be  written  so  the  risks  (of  not  meeting  the  plans)  can 
be  determined 
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Risk  Management  Example 


NORTHROP  GRUMMAN 


Risk  Management  -  Specific  Practice  1.1 
Determine  risk  sources  and  categories . 

x  What  sources  of  risk 
should  we  consider  in 
identifying  risk? 

•  Frequently  heard: 

-  “The  sources  are  the  same  as  the  categories” 

-  “Anybody  can  identify  a  risk  at  any  time” 

-  “Risks  are  categorized  by  impact:  cost,  schedule,  performance” 

•  Explaining  the  value 

-  Setting  expectations  for  the  types  of  risk  to  consider  helps  ensure  that 
ALL  risks  get  identified 

-  How  the  risks  will  be  categorized  helps  ensure  they  are  managed 
appropriately 


How  should  we  categorize 
the  risks  once  they  are 
identified? 

•Urgency 

•Assignee 
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SEI  Risk  Taxonomy 


NORTHROP  GRUMMAN 


A.  Product  Engineering 


B.  Development  Environment  C.  Program  Constraints 


1.  Requirements 

a.  Stability 

b.  Completeness 

c.  Clarity 

d.  Validity 

e.  Feasibility 

f.  Precedent 

g.  Scale 

2.  Design 

a.  Functionality 

b.  Difficulty 

c.  Interfaces 

d.  Performance 

e.  Testability 

f.  Hardware 

g.  Non-Developmental  SW 

3.  Code  and  Unit  Test 

a.  Feasibility 

b.  Testing 

c.  Coding/Implementation 

4.  Integration  and  Test 

a.  Environment 

b.  Product 

c.  System 

5.  Engineering  Specialties 

a.  Maintainability 

b.  Reliability 

c.  Safety 

d.  Security 

e.  Human  Factors 
16  f.  Specifications 


1.  Development  Process 

a.  Formaility 

b.  Suitability 

c.  Process  Control 

d.  Familiarity 

e.  Product  Control 

2.  Development  System 

a.  Capacity 

b.  Suitability 

c.  Usability 

d.  Familiarity 

e.  Reliability 

f.  System  Support 

g.  Deliverability 

3.  Management  Process 

a.  Planning 

b.  Project  Organization 

c.  Management  Experience 

d.  Program  Interfaces 

4.  Management  Methods 

a.  Monitoring 

b.  Personnel  Management 

c.  Quality  Assurance 

d.  Configuration  Mgmt 

5.  Work  Environment 

a.  Quality  Attitude 

b.  Cooperation 

c.  Communication 

d.  Morale 


1 .  Resources 

a.  Schedule 

b.  Staff 

c.  Budget 

d.  Facilities 

2.  Contract 

a.  Type  of  Contract 

b.  Restrictions 

c.  Dependencies 

3.  Program  Interfaces 

a.  Customer 

b.  Associate  Contractors 

c.  Subcontractors 

d.  Prime  Contractor 

e.  Corporate  Management 

f.  Vendors 

g.  Politics 


Taxonomy-Based  Risk  Identification,  ” 
Marvin  J.  Carr,  et  al,  Software 
Engineering  Institute,  CMU/SEI-93- 
TR-6,  ESC-TR-93-183,  June  1993. 
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Level  4/5 


NORTHROP  GRUMMAN 


•  The  value  of  Level  4/5  is  insight 

-  Level  4/5  is  1 0-20%  cheaper  than  Level  3 
(even  though  more  is  being  done 

-  Quantitative  management  establishes 
expected  ranges  of  process  performance 

-  Process  are  stable  and  predictable  -  unusual  process  behaviors  can 
be  quickly  identified,  so  effective  corrective  action  can  be  taken 

•  To  realize  the  value  of  Level  4/5 

-  Processes  have  to  be  stable  (performed  consistently) 

-  Processes  under  statistical  control  must  support  business  objectives 

-  Data  has  to  be  useful  and  clean 

-  Analysis  has  to  lead  to  actions 
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How  Does  Level  4  &  5  Benefit  the 
Customer? 


NORTHROP  GRUMMAN 
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Organizational  Process 
Performance 


Quantitative  Project 
Management 


Organizational 

Performance 

Management 


Causal  Analysis  and 
Resolution 


c 


More  accurate  estimates 


Problem  behaviors  are 
recognized  faster,  enabling 
quicker  resolution 


The  project  benefits  from 
improvements  found  and 
proven  on  other  projects 

. I . « . — . f . f . 

The  project  fixes  the  source  of 
defects  to  prevent  future 
defects 

//  aW 


“How  Does  High  Maturity  Benefit  the  Customer?”,  R.  Hefner, 
Systems  &  Software  Technology  Conference,  2005 
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Quantitative  Management  Example 


NORTHROP  GRUMMAN 


Suppose  your  project  conducted 
several  peer  reviews  of  similar 
code,  and  analyzed  the  results 

Mean  =  7.8  defects/KSLOC 
+3a  =  11 .60  defects/KSLOC 
-3a  =  4.001  defects/KSLOC 


•  What  would  you  expect  the  next 
peer  review  to  produce  in  terms 
of  defects/  KSLOC? 

•  What  would  you  think  if  a  review 
resulted  in  10  defects/KSLOC? 

•  3  defects/KSLOC? 


Observation  Number 


UCL=  11.60 


Mean=7.8 


LCL=4.001 


“A  Practical  Guide  to  Implementing  Levels 
4  and  5  (tutorial),”  R.  Hefner,  2006  CM  Ml 
Technology  Conference  and  User  Group 
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Summary 


NORTHROP  GRUMMAN 


•  Convincing  someone  to  implement  a  CMMI  practice  is  easier 
when  you  can  explain  how  the  CMMI  practices  add  value 

•  Insight  comes  from  observing  practical,  value-added 
implementations 
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Resilient  Service:  CMMI  -SVC 
and  CERT-RMM 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

November  201 1 

Eileen  Forrester,  Richard  Caralli 
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Carnegie  Mellon 
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What  we  will  cover 


•  An  alternate  way  to  get  some  security  coverage 

•  What  is  resilient  service 

•  CMMI-SVC  and  RMM 


•  Quality  and  mission  assurance 

•  An  example  resilient  service  using  both  models 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 

©2011  Carnegie  Mellon  University 
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Assembling  a  multi-model  approach  to  improving  service  quality 
and  ensuring  service  resilience  in  complex  risk  environments 

It  A  j  4/7. 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 

©2011  Carnegie  Mellon  University 


Improving  Service  Management 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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Why  Should  We  Fill  the  Gap? 


Completeness  of  Improvement  Journey 

•  Organizations  have  business  problems  to  solve  that  cross  model 
boundaries 

•  Framing  these  issues  in  a  common  language  helps 

Appraisal  or  Audit  or  Compliance  Need 

•  Organizations  with  multiple  accreditations  are  faced  with  frequent 
internal  audit  and  appraisal  issues 

•  One  common  framework  cuts  appraisal  and  audit  costs  &  minimizes 
disruption  to  busy  front  line  workers 

Model  Completeness 

•  Security  issues  are  not  “additional”  to  service  delivery  they  are  integral 
to  it 


— Forrester  CMMI  &  Results 

=-^=-  Software  Engineering  Institute  Carnegie  Mellon  April  2011 
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How  To  Fill  The  Gap? 


RMM? 

•  Lots  of  great  material 

•  High  specification  of  how  to  solve  security  questions 

•  Probably  interpreted  in  some  people’s  minds  as  “An  Extra  Model  to 
adopt!” 

Services  PA 

•  Services  security  content  needs  steward  approval 

CMMI-SVC  “Pseudo  PA”  Material 

•  Quick 

•  Seed  for  further  development 

•  Small  scale  addition  to  existing  model 


— Forrester  CMMI  &  Results 

=-^=-  Software  Engineering  Institute  Carnegie  Mellon  APril2011 
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Developing  a  “Bolt  on”  for  CMMI 


Requirements 

•  Needs  to  work  with  other  CMMI  process  areas 

•  Needs  to  have  fit  CMMI  architecture 

-  Required  Components 

-  Expected  Components 

-  Informative  Material 

•  Generic  Practices 

•  Specific  Material 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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GP  Relationship  -  Conclusions 

ISO  27001  clauses  are  short  statements  of  requirements 

•  Not  much  detail 

•  No  “informative  material”  -  example  work  products,  etc. 

ISO  27001  -  Is  less  explicit  on  Stakeholder  Management 
Using  CMMI  GPs  would 

•  Further  help  embed  good  practice 

•  Build  upon  existing  material 


— Forrester  CMMI  &  Results 
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ISO  27001  -  Establishing  ISMS 


Clause  4.2.1  -  Establish  the  Information  Security  Management  System 

•  Scope  the  security  system 

•  Define  an  approach  to  identifying  and  evaluating  security  threats 

•  Define  how  to  deal  with  them 

•  Obtain  management  approval  for  the  plans  and  mechanisms  defined 


Software  Engineering  Institute 


Carnegie  Mellon 
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ISO  27001  -  Put  the  ISMS  in  Place 


Clause  4.2.2  -  Implement  and  Operate  the  Information  Security 
Management  System 

•  Instigate  a  plan  to  operate  the  security  system 

•  Manage  the  level  of  threat. 

Clause  4.2.3  -  Monitor  and  Review  the  ISMS 

•  Use  ISMS  mechanisms  to  monitor  threats 

•  Take  action  to  address  threats 

Clause  4.2.4  -  Maintain  and  Improve  the  ISMS 

•  Measuring  and  monitor  the  system 

•  Implement  corrections  or  improvements 


— Forrester  CMMI  &  Results 
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Security  Pseudo  PA  -  Basic  Structure 


Examination  of  ISO  27001  provided  a  nice  suggestion  of 
initial  content 

•  Establish  and  Maintain  a  Security  Management  System 

•  Use  the  Agreed  Security  Management  System  to  Provide  Required 
Security 

•  Note  we  dropped  “information”  in  our  version 

Under  these  two  strands  we  can  construct  statements  that 
look  and  feel  like  practice  statements 

•  Ideal  for  appraisal  purposes 

•  Very  valuable  for  improvement  teams  constructing  an  improvement 
plan 

•  One  language  style,  one  plan,  potentially  multiple  models  engaged 


— Forrester  CMMI  &  Results 

=-^=-  Software  Engineering  Institute  Carnegie  Mellon  April  2011 
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Pseudo  PA: 

Security  Management  (SM) 

ESG1  -  Establish  a  Security  Management  System 

•  ESP1.1  Establish  Security  Objectives 

•  ESP1 .2  Establish  an  Approach  to  Threat  Assessment 

•  ESP1 .3  Identify  Security  Threats 

•  ESP1 .4  Evaluate  and  Prioritize  Security  Threats 

•  ESP1.5  Establish  a  Security  Management  Plan 

•  ESP1 .6  Obtain  Commitment  to  the  Security  Management  Plan 

ESG2  -  Provide  Security 

•  ESP2.1  Operate  the  Security  Management  System 

•  ESP2.2  Monitor  the  Security  Management  System 


— Forrester  CMMI  &  Results 
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Framework  For  Building  Upon 


Basic  Pseudo  PA 
Architecture 


Ability  to 
Appraise 


CMMI  is  used  for  more  than 
appraisals,  what  about  the 
implementation  and  improvement 


— Forrester  CMMI  &  Results 
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Informative  Material 


Informative  Material  provides: 

•  Subpractices 

•  Notes 

•  Examples 

•  Elaborations 

•  Example  Work  Products 

•  Etc. 

All  these  help  the  implementation  of  good  practice 

This  PA  is  quite  general,  so  RMM  is  also  a  source  for  more  detail  and 
rigor. 


— Forrester  CMMI  &  Results 

Software  Engineering  Institute  Carnegie  Mellon  APril2011 


)2011  Carnegie  Mellon  University 


Example  New  Informative  Material 


ESP1.2  Establish  an  Approach  to  Threat  Assessment 

Establish  and  maintain  an  approach  to  assessing  vulnerabilities  and 
threats  to  essential  assets. 

Subpractices 

1.  Select  methods  for  assessing  security  threats 

2.  Define  criteria  for  evaluating  and  quantifying  security 
threats. 

3.  Describe  responsibility  and  resources  for  evaluating 
vulnerabilities  and  threats. 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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Next  Moves 


Pseudo  PA  has  been  tested  on  a  number  of 
appraisals 

Challenge  to  develop  more  “PA”  like  substructure 

•  Practices 

•  Subpractices 

•  Example  work  products 

•  GP  Elaborations 

We  have  made  a  start-but  now  would  like  to 
engage  a  wider  audience  to  take  the  discussion 
forward 


— Forrester  CMMI  &  Results 
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Community  Feedback  and  Input 


Should  this  work  be  taken  further? 

Is  the  scope  useful  for  improvement? 

What  could  be  done  next  to  make  it  more  credible? 
We  would  like  your  comments. 

•  cmmi-comments@sei.cmu.edu. 


Software  Engineering  Institute 


Carnegie  Mellon 
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Some  Useful  Links 


CMMI  for  Services  Model 

http://www.sei.cmu.edu/cmmi/tools/svc/index.cfm 

CMMI  for  Services  and  Security  Whitepaper 

http://www.sei.cmu.edu/cmmi/tools/svc/upload/Securitv- 

and-CMMI-SVC.pdf 

CMMI  for  Services  Book 

http://www.amazon.com/CMMI-Services-Guidelines- 

Superior- 

Enqineerinq/dp/0321711521/ref=sr  1  1?ie=UTF8&aid=1 

30441 5568&sr=8-1 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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Summary  on  the  Pseudo  PA 


IS020000,  ITIL,  &  CMMI  all  work  very  well  together 

CMMI  misses  one  component  in  common  with  the  other  approaches: 
security 

ISO  27001  provided  a  starting  point  for  developing  a  “pseudo”  process 
area:  SM 

We  are  seeking  community  input  to  develop  this  pseudo  process  area 
further 


Software  Engineering  Institute 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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How  Resilient  Am  I?  - 1 


When  asked: 

•  How  resilient  am  I? 

•Am  I  resilient  enough? 

•  How  resilient  do  I  need  to  be? 


How  Resilient  Am  I?  -  2 


•  Do  I  need  to  worry  about  operational 
resilience? 


•  If  services  are  disrupted,  will  it  make 
the  news?  Will  I  end  up  in  court?  in 
jail?  Will  I  be  able  to  stay  in  business? 

•  Do  I  meet  compliance  requirements? 

•  How  resilient  am  I  compared  to  my 
competition? 

•  Do  I  need  to  spend  more  $$  on 
resilience?  If  so,  on  what? 

•  What  am  I  getting  for  the  $$  I’ve 
already  spent? 


— Forrester  CMMI  &  Results 

=-^=-  Software  Engineering  Institute  Carnegie  Mellon  April  2011 
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What  is  CMMI? 

The  Capability  Maturity  Model  Integration  (CMMI) 

•  is  a  framework  for  management  practices 

•  provides  organizations  with  the  essential  elements  of  effective  processes 
that  improve  performance 

•  can  be  used  as  a  benchmark,  but  is  about  quality  improvement 


CMMI 
for  Services 

GuJ  dtlum 

«k 

Service 

.  V  • 

r.  i L"  n  £1.  Fumudir 

lhlll»l| 
Shnim 

s.-.ikk  shrum 


CMMI  for  CMMI  for  CMMI  for 

Development  Services  Acquisition 

(CMMI-DEV)  (CMMI-SVC)  (CMMI-ACQ) 


CMMI 

for  Development 


Integration 
and  Product 
Improvement 


CMMI 

for  Acquisition 


Guidelines 
for  Improving 
the  Acquisition 
of  Products 
and  Services 


The  CMMI  Product  Suite  is  a  set  of  CMMI-related  products  that 
includes  CMMI  models,  appraisal  method,  and  CMMI  training 
courses. 


— Forrester  CMMI  &  Results 
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Relationships  Among  CMMI  Models 


Shared  PA  (SAM) 


Development-specific  PAs 


Service  “addition”  PA  (SSD) 
Service-specific  PAs 


Core  PAs 

Include  model-specific 
informative  material 


Acquisition-specific  PAs 


— Forrester  CMMI  &  Results 
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A  Look  at  CMMI-SVC 


Shared  PA 
(SAM) 


( 

)  CMMI-SVC 

t 

a 

Services-specific  PAs 
*CMMI-SVC  addition 


Core  PAs 

Include  service-specific 
informative  material 


CMMI-ACQ 


/ 


Capacity  and 

Availability 

Management 


Service 

Continuity 


Strategic  Service 
Management 


Service  System 
Development 


Software  Engineering  Institute 


- : - \ 

Define,  and  Establish,  and  Deliver  Services 


Monitor  and  Control  Service  and 
Work  Products 


Ensure  Service  Mission  Success 


Make  Work  Explicit  and  Measurable 


Incident  Resolution 
&  Prevention 


Manage  Decisions,  Suppliers,  and 
Standard  Services 


Service 

Delivery 


Service  System 
Transition 


Create  a  Culture  to  Sustain 
Service  Excellence 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 
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What  is  CERT®-RMM? 


CERT-RMM  is  a  capability 
model  for  managing  and 
improving  operational  resilience. 


-  Software  Engineering  Institute 


•  Guides  implementation  and 
management  of  operational 
resilience  activities 

•  Converges  key  operational 
risk  management  activities: 
security,  BC/DR,  and  IT 
operations 

•  Defines  maturity  through 
capability  levels  (like  CMMI) 

•  Improves  confidence  in  how 
an  organization  responds  in 
times  of  operational  stress 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
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CERT-RMM  in  the  life-cycle 


Operational  resilience  management  focuses  on  the  deploy,  operate,  and 
decommission  phases,  but  reaches  back  to  development  phase  of  lifecycle 
to  ensure  consideration  of  security  and  continuity  issues  prior  to  placing 
assets  in  production. 


om- 


Design 


Develop 


Acquire 


Deploy  2  Operate 
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Operational  resilience 


Resilience:  The  physical  property  of  a 
material  when  it  can  return  to  its  original 
shape  or  position  after  deformation  that 
does  not  exceed  its  elastic  limit 

[wo  rdnet.princeton.edu] 


Operational  resilience:  The 
emergent  property  of  an 
organization  that  can  continue 
to  carry  out  its  mission  after 
disruption  that  does  not  exceed 
its  operational  limit 

[CERT-RMM] 
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Services  in  CERT-RMM 


The  resilience  of  high-value  services  ensures  the  resilience  of  the 

mission. 


Service  resilience  is  a  factor  of  asset  resilience — if  an  asset  is 
disrupted  or  fails,  the  service  may  suffer. 


Service  resilience  is  the  object  of  CERT-RMM  processes. 
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Assets 


Something  of  value  to  the  organization 
Used  by  business  processes  and  services 

CERT-RMM  focuses  on  four  types: 


People 


Information 


Technology 


Facilities 
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Organizational  Context 
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CERT-RMM  &  CMMI  in  the  life  cycle 


own 


Design 


Develop 


Acquire 


CMMI-DEV 


CMMI-ACQ 


CMMI-SVC 


DEVELOPMENT 


Deploy  ^  Operate  J  Decommision 


CERT-RMM 


OPERATION 
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CERT-RMM  architectural  elements 


•  26  process  areas 

•  Arranged  in  a  continuous 
representation 

•  Goals,  practices,  sub-practices,  and 
work  products  that  specifically 
define  each  process  area 

•  Goals,  practices,  and  sub-practices 
that  generically  define  increasing 
levels  of  capability 

•  Implementation  and  adoption 
examples 

•  An  appraisal  methodology  to 
determine  capability  levels 


CERT-RMM  uses  proven 
architectural  elements  of 
CMMI  and  applies  them  in 
an  operational  context. 
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CERT-RMM  at  a  glance 


Engineering 


ADM 

Asset  Definition  and  Management 

CTRL 

Controls  Management 

RRD 

Resilience  Requirements  Development 

RRM 

Resilience  Requirements  Management 

RTSE 

Resilient  Technical  Solution  Engineering 

SC 

Service  Continuity 

Enterprise  Management 

COMM 

Communications 

COMP 

Compliance 

EF 

Enterprise  Focus 

FRM 

Financial  Resource  Management 

HRM 

Human  Resource  Management 

OTA 

Organizational  Training  &  Awareness 

RISK 

Risk  Management 

Operations  Management 


AM 

Access  Management 

EC 

Environmental  Control 

EXD 

External  Dependencies 

ID 

Identity  Management 

IMC 

Incident  Management  &  Control 

KIM 

Knowledge  &  Information  Management 

PM 

People  Management 

TM 

Technology  Management 

VAR 

Vulnerability  Analysis  &  Resolution 

Process  Management 

MA 

Measurement  and  Analysis 

MON 

Monitoring 

OPD 

Organizational  Process  Definition 

OPF 

Organizational  Process  Focus 

26  Process  Areas  in  4  categories 
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Enterprise  management 


Seven  process  areas  that 
support  the  resilience 
management  process 


Governance,  Risk,  &  Compliance 


COMP 


uli  nli  nib 


RISK 


Supporting  Resilience 
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FRM  1 

| 

lib 

nib  nib  nib  nib 


HRM  1 

|l| 

H 

OTA 


Software  Engineering  Institute  Carnegie Mellon  APril2011 

—  °  ©2011  Carne 


Forrester  CMMI  &  Results 
April  2011 

©2011  Carnegie  Mellon  University 


Engineering 


Six  process  areas  for 
establishing  resilience 
for  organizational 
assets,  business 
processes,  and 
services 


Software  Engineering 


Asset  Management 


Requirements  Management 


RRD 


43  43 


RRM 


Establishing  and  Managing  Resilience 


RTSE 


A> 
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CTRL 


Operations  management 


Nine  process  areas  for 
managing  the  operational 
aspects  of  resilience 


Asset  Resilience  Management 


EC 

KIM 

PM 

TM  I 

#% 

#% 

Threat,  Incident,  &  Access  Management 


AM 

ID 

IMC 

VAR  I 

Supplier  Management 
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Process  management  process  areas 


Four  process  areas  for  defining, 
planning,  deploying, 
implementing,  monitoring, 
controlling,  appraising, 
measuring,  and  improving 
operational  resilience 
management  processes 


Data  Collection  &  Logging 


MON 


o 


Process  Management 


MA 


OPD 


o  o  o 


OPF 
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Positioning  CERT-RMM  with  CMMI 


SCAMPI-based  appraisal  methods 


P-CMM 


Uses  Process  Areas  from  Core  and  CMMI-DEV 
Shares  connection  in  Service  Continuity  (SCON) 
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CMMI-DEV 


CERT-RMM  and  CMMI-SVC 


CERT-RMM 


► 


Expands  SCON  to  resiliency  perspective 


Shares  an  organizational  focus,  rather  than  project 


Focus  is  on  high-quality  service  delivery  that  is  resilient 


Model  use  will  identify  additional  synergies 
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A  service  example:  US  auto  insurance 

Olive  Vehicle  Insurance  (OVIG)  provides  car  and  light  truck  insurance. 

Customer  services  include  providing  quotes,  issuing  policies,  billing  and 
processing  premiums,  processing  claims,  providing  legal  services,  and 
providing  vehicle  repair. 


They  pride  themselves  on  being  easy  to  reach  and  quick  to  act  when 
the  customer  needs  them.  They  are  facing  an  increasingly  demanding 
regulatory  environment  in  the  US. 


What  does  it  mean  for  these  services  to  be  resilient?  What  assets  must 
be  resilient?  What  practices  in  RMM  go  beyond  RSKM,  IRP,  and 


SCON? 
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CMMI-SVC  PAs  that  ensure  mission  success 


Incident  Resolution  and  Prevention  (IRP): 

handling  what  goes  wrong — and  preventing  it  from  going  wrong  ahead  of  time  if  you  can 

Risk  Management  (RSKM): 

supporting  the  success  of  your  service  mission  by  anticipating  problems  and  how  you  will  handle 
them — before  they  occur 

Service  Continuity  Management  (SCON): 

being  ready  to  recover  from  a  disaster  and  get  back  to  delivering  your  service 

Service  System  Transition  (SST): 

getting  new  systems  in  place,  changing  existing  systems,  and  retiring  obsolete  systems,  all  while 
making  sure  nothing  goes  terribly  wrong  with  service 
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CMMI-SVC  PAs  taken  further  with  RMM  PAs 


Incident  Resolution  and  Prevention  (IRP): 

IMC  is  obvious,  but  also  VAR  in  RMM  goes  further  than  goal  3  in  IRP  to  actively  watch 
and  resolve  vulnerabilities  before  they  become  incidents  that  disrupt  insurance  services 

Risk  Management  (RSKM): 

KIM  practices  can  be  used  to  apply  controls  for  confidentiality,  integrity,  and  availability 
to  critical  data,  such  as  customer  information 

CTRL  practices  go  further  to  applying  controls  to  service  processes  such  as  paying 
claims,  so  that,  for  example,  no  claim  is  paid  twice  and  that  claim  data  is  kept 
confidential  and  not  accidentally  modified 

Service  Continuity  Management  (SCON): 

SC  in  RMM  explodes  the  goals  and  practices  found  in  SCON  with  considerably  more 
detail;  for  example,  a  data-intensive  service  like  insurance  can  find  more  advice  on 
managing  effects  on  vital  records;  in  addition,  SC  makes  clear  the  distinctions  among 
continuity,  recovery,  and  restoration  of  service 

Also  consider: 

EXD,  which  goes  further  than  SAM  to  further  resilience,  more  info  on  external 
dependencies  and  service  agreements 

MON,  which  goes  beyond  MA  in  SVC  to  have  “feelers”  out  for  data  so  that  the 
organization  knows  how  their  data  stands  relative  to  threats  and  vulnerabilities 
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Summary 

GPs  and  Pseudo  PA  approach  allows  you  to  selectively  borrow  from 
additional  models,  even  during  appraisal. 

RMM  and  CMMI-SVC  combination: 

•  The  goal  of  CMMI-SVC  is  equip  organizations  to  improve  processes  and  ensure  high- 
quality  service  management  and  delivery  at  an  affordable  cost. 

•  The  goal  of  CERT-RMM  is  to  improve  processes  to  ensure  that  essential 
organizational  services  meet  their  mission  consistently  in  the  face  of  shifting 
operational  risk. 

•  They  share  common  content,  similar  product  suites  to  support  use,  and  provide 
different  detail  and  specificity  that  you  can  choose  from  to  meet  your  precise  needs. 

•  These  two  models  are  being  combined  in  appraisal  and  implementation. 

•  In  short,  CMMI-SVC  and  CERT-RMM  are  synergistic  and  amenable  to  a  continuous 
approach  based  on  your  business  needs  for  resilient  service. 
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CERT -RMM  contacts 


Rich  Caralli 

RMM  Architect  and  Lead  Developer 
rcaralli@cert.org 


Lisa  Young 

RMM  Appraisal  Lead  and  Developer 
lry@cert.org 


Richard  Lynch 

Public  Relations  — All  Media 
Inquiries 

public-relations@sei.cmu.edu 


Joe  McLeod 

For  info  on  working  with  us 

jmcleod@sei.cmu.edu 


Software  Engineering  Institute 


David  White 

RMM  Transition  Lead  and  Developer 
dwhite@cert.org 


Julia  Allen 

RMM  Developer/Measurement  Team 
Lead 

jha@sei.cmu.edu 


SEI  Customer  Relations 

customer-relations@sei.cmu.edu 

412-268-5800 


http://www.cert.org/resilience/ 
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Contact  information 


Eileen  Forrester 

Manager,  CMMI  for  Services 
SEPM 

Telephone:  +1  412-268-6377 
Email:  ecf@sei.cmu.edu 


Web 

www.sei.cmu.edu/cmmi 

www.sei.cmu.edu 


Software  Engineering  Institute 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 


Email: 
Telephone: 
SEI  Phone: 
SEI  Fax: 


info@sei.cmu.edu 

+1  412-268-5800 
+1  412-268-5800 
+1  412-268-6257 


Carnegie  Mellon 


Forrester  CMMI  &  Results 
April  2011 

©2011  Carnegie  Mellon  University 


NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE  MATERIAL 
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Backup  slides  as  needed 
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Imperatives  for  building  CERT-RMM 


Cultural  shifts 


Increasingly  complex  operational 
environments;  traditional  approaches  failing 

Silo  nature  of  operational  risk  activities;  a 
lack  of  convergence 

Lack  of  common  language  or  taxonomy 

Overreliance  on  technical  approaches 

Lack  of  means  to  measure  organizational 
capability 

Inability  to  confidently  predict  outcomes, 
behaviors,  and  performance  under  times 
of  stress 
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How  Resilient  Am  I? 


Software  Engineering  Institute 


3 


What  should  I  be 
measuring  to  determine 
if  I  am  meeting  my 
performance  objectives 
for  resilience? 


What  is  the  business 
value  of  being  more 
resilient? 
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Organizational  context  -  disruption 


Operational  risk  can  disrupt  an  asset 


And  lead  to  organizational  disruption 
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CERT-RMM  links  to  codes  of  practice 


- - 

\ 

Codes  of  Practice: 

BS25999-1:2006 
CMMIvl.2 
CMMI  for  Services 
CobiT  4.1 
COSO  ERM 
DRII  GAP 

FFIEC  Handbooks  (Security, 
BCP) 

ISO  20000-1 :2005(E) 

ISO  20000-2:2005(E) 

ISO  24762:2008(E) 

ISO  27001:2005 
NFPA1600  (2007) 

/ 

PCIDSSvl.1  / 
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How  Resilient  Am  I?  - 1 


When  asked: 

•  How  resilient  am  I? 

•Am  I  resilient  enough? 

•  How  resilient  do  I  need  to  be? 


How  Resilient  Am  I?  -  2 


•  Do  I  need  to  worry  about  operational 
resilience? 


•  If  services  are  disrupted,  will  it  make 
the  news?  Will  I  end  up  in  court?  in 
jail?  Will  I  be  able  to  stay  in  business? 

•  Do  I  meet  compliance  requirements? 

•  How  resilient  am  I  compared  to  my 
competition? 

•  Do  I  need  to  spend  more  $$  on 
resilience?  If  so,  on  what? 

•  What  am  I  getting  for  the  $$  I’ve 
already  spent? 
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